Property based testing
You will always hear that writing tests is a good thing. But are you 100% sure about the pertinence of your test suite? What if you test only edges cases and miss something? Property based testing let you generate your tests instead of writing them. Let’s see what this is, how to use it, and how we can leverage it to write robust UI tests.
Let’s say for the sake of the demonstration that we want to implement a sort algorithm in our application.
Our implementation looks like this:
If we want to test our code, we could write some example based unit tests like so:
Attentive readers may have spotted that our code is buggy, but our tests pass anyway. How could we test it better to find the bug?
Property based testing
We wrote some tests, that’s a very good thing, but our test suite is really minimal. In a perfect world, we would like to test the
quickSort() function for every array of integers.
What we call a property based test is a test that assert a certain property holds for every input we use (meaning every array of integers in our case).
Conceptually, here is a property based test:
In our example, the property is rather simple: if we use
quickSort() on an array, the array should be sorted (meaning every element is lower or equal than the next element).
In pseudo code, here is our property based test.
In practice, even if this is not possible to test every array of integer in the world, we can write tests that run this property for a lot of arrays. And this will be more powerful that the first example based tests we wrote earlier.
The most known library to do property based testing is called QuickCheck in Haskell. In Swift, we can use SwiftCheck instead.
Here is the protocol at the core of the library:
In essence, this protocol is composed of two components:
- a random value generators (primitive types already conform to
- a way to shrink a value from an instance
I encourage you to go read the tutorial playground if you want to well understand the concepts.
The random value generator is useful to generate a lot of random inputs, but what do we call shrinking?
When SwiftCheck finds a test that fails for a random input, it will not simply return the input, but will iterate on some shrunk values of this input to find the minimal input that make the test fail. By definition, shrunk values have a size less or equal than the initial value.
For instance, if we ask the shrunk values for the
"test" string, we will get this result:
Back to the example
Now that we understand how to implement a property based test, let’s do it for the
quickSort() function we defined earlier.
Following the SwiftCheck syntax, here is the test:
When we run this code, we get an error right away !
Our tests were not that strong because we already have found a failing input… In our case this is the array
Note that the test did fail first with the input
[-5, 1, 5, 4]. Then the library did shrink the value
[-5, 1, 5, 4] to find the minimal input that makes our property fail. It generated more than 600 different values (7 tests and 11 shrinks) to find that
[1, 0] is the minimal input that makes the test fail.
We can easily spot the bug with the failing input.
This time the result is correct, all the 100 tests pass.
When to use property based testing?
The advantages of property based testing are:
- they are more general and can replace many example based tests
- they test all the edge cases (null, negative numbers, empty arrays, uncommon strings)
- they are reproducible (once a test fail, we know for which input, and we can rerun the test)
- they shrink the input in case of a failure
But that does not mean you have to remove all your example based tests and replace them with property based tests. In practice it can be really difficult to write property based tests, and example based tests are important because they are simple. Meaning that someone else can quickly understand their utility.
There are some special cases where property based testing shines. For instance, when you have two symmetrical functions (for instance an
Encoder and a
Decoder): for any input, the result of passing the input in the first function, then in the reverse function should be equal to the initial input.
This idea is well covered in the talk of Jack Flintermann, creator of the Dwiff library that explains that property based testing helped him catch some weird bugs during refactoring.
For example, let’s say we want have a function that sub divide an array into chunks of a size
The reverse function is easy to find, we just need to reassemble all the arrays to get the input back.
The test associated with this function could be:
What about UI tests?
As we mainly write code that is related to UI, how could we leverage property based testing to test our layouts? For instance, it’s very common to forget activating an
NSLayoutConstraint, or to provide the wrong
constant value, and this results in layout issues. And sometimes these issues do not appear during the development phase, but later, in production with real data.
To use property based testing, the idea is similar to our previous example. We need to generate random inputs and find a property that holds for our layout for every input.
The property is easy to find: a view is just a bunch of subviews, and we want to assert that our layout is correct when there are no views overlap, no autolayout errors and no clipped subviews.
Now, how to generate our inputs?
I always create a struct called a
ViewModel to configure my views. This is just a dumb data structure that holds all the properties that will be displayed in the view (booleans to hide / show subviews, strings to display in labels, etc.).
So all I need is to generate random view models, pass them to my view, and then assert that the layout is correct.
Here is an example of how to generate a random view model:
We now have a random
If you want to print a random instance of a
ViewModel, you can do:
Now testing our layout is straightforward:
The implementation of the methods
hasNoFrameOverlap is left as an exercise to the reader, but some time ago, LinkedIn created a library heavily inspired by this approach (even so they do not provide real random values). You can find the implementations on the repository.
We have seen how powerful property based testing is. Instead of writing one or two example based tests per feature, it allows you to generate thousands of random tests very easily.
The drawback is that this is not always straightforward to find a property that holds for every test you want to write. So even if you can’t use this technique right now, keep it in mind the next time you need it.