The many types of tests in software testing

The content here is under the Attribution 4.0 International (CC BY 4.0) license

When deciding that testing is the right decision and it needs to be done, comes the second big issue, what kind of test do I need to do? This section aims to clarify each type of test and how to apply them, Martin Fowler wrote a good article and explains in his first paragraph:

Its essential point is that you should have many more low-level unit tests than high level end-to-end tests running through a GUI.

The pyramid defines that UI tests are expensive in their execution and unit tests are faster. However, as we will see here, the definition of the type of tests varies based on the author and their interpretation. Let’s start with Santos et al definition of different types of tests.

For them, an integration test is defined as the test of a repository and the database. Or, the integration with a third-party service. The following images are reproduced from their book:

In the same chapter, Santos et al. also mention the lack of standardization about each type of test. The problem with the lack of a proper standard is also reflected here. In the content that follows the attempt is to bring different perspectives on the different types os tests and then share a common picture of each one of them.

End to end

The end-to-end testing is the one that faces business requirements and verifies what the customer who is waiting for the software wants the system to do.

Acceptance testing

Is the type of test that is closest to the business facing rather than the technical aspect of it. It exercises most of the software in question but it will lack some of its parts. The part missing is replaced by a test-double or the test itself ignores it to focus on the feedback rather than on the completeness of the system parts.

Black-box

The type of test that is done when there is no internal knowledge of the system. The interaction of the test is the same as the normal user has.

White-box

The test at hand is performed with knowledge of its internal functionality.

Load testing

Aim at adding load into the system to meet requirements that are usually treated as non-functional requirements, such as performance, security and reliability.

• Load testing

Contract testing

Contract testing focuses on the contract level between two applications. One is the consumer and the other is the provider. Consumers base their code on complying with the specified contract, and this feedback the providers with the statistics of who is using this contract.

Providers benefit from the contract as it will let them know if the changes done will affect consumers and break their application. Contract testing provides feedback for developers to maintain a backward-compatible flow.

• Explore contract testing

Integration tests

Sommerville defines integration testing as the testing of the complete system. Such a test is used to verify if the software requirements were met. If so, the software is ready to be delivered to the customers [2], Black shares the same definition as Sommerville [1].

Santos et al. shared a different perspective on the type of integration test. The scope is reduced to a specific part of the system. More recently, Aniche suggested the same type of understanding about integration testing [3]:

;

If we were to compare the definition of Sommerville with Santos et al. The vision that Sommerville has fits in the End-to-end test of Santos et al. Another way of thinking about integration tests from this point of view, is to integrate the type of request that comes in first. in the image we see that the first contact with the user is from a view, therefore, this could be an integration itself. We might want to test the same logic with a Rest API, GraphQL or any other communication protocol.

Unit testing

Nowadays unit test is (in my opinion) the most used one, and the big reason is that it provides a way to test the unit, which means a very small piece of your application. Is hard to think in a unit when we have to handle HTTP requests, query the database or even validate the data coming, but besides that unit test is the easiest to get started. In object-oriented programming, a unit means a method. A method is the behavior that an object has and itself can be considered a unit. The interaction of objects is common as well, but for units, we should guarantee that only one thing will be tested at a time. Aniche shares the same definition, as he bases his definition on the test pyramid, the following image reproduces as is, the definition at hand:

The simplest example we can demonstrate here is a mathematical operation, where we would have a class called Math and for each operation a method.

class Math
{
    public function sum($a, $b)
    {
        return $a + b;
    }
}

And its corresponding test class:

class MathTest extends PHPUnit
{
    public function should_sum_two_numbers()
    {
        $math = new Math();
        $this->assertEquals($math->sum(5, 5), 10);
    }
}

There is a gray area among practitioners who focus on the unit test with a single class. Having a test class and a production class might point to a weak testing structure. Starting with this approach is not a problem, however, in projects that are maintained in production, such structure reveals a high level of unneeded coupling. Often, those test suites are changing without changing their behavior. The change comes just to comply with something that is related to style or refactoring. When that is the case, further analysis should be done in a data-driven fashion to better understand the coupling between test code and production code.

Through years of experience writing test code and working in the industry for medium to large projects, I reflected on my experience of defining unit testing:

Unit testing in my opinion is the basis of the automated testing that can be written before or after the production code. I like to make definitions explicit as in professional projects, people who write software often mix unit testing with test-driven development. Both are closely related but are not the same. The principal differentiator of both is precisely the moment that the test is created. For test-driven development, the test must be created before. The unit test alone has no such restriction.

Property based testing

Property-based testing helps developers to make their assumptions explicitly in the code and handle edge cases that usually are not taken into account, even with a code that uses a test-first approach. Thomas and Hunt exemplify this case with a sorting algorithm [4]. This type of test is also used by Aniche in his approach of a systematic point of view in tackling testing activities [3].

Discussion

References

1. [1]R. Black, Pragmatic software testing: Becoming an effective and efficient test professional. John Wiley & Sons, 2016.
2. [2]I. Sommerville, “Software Engineering 10th Edition (International Computer Science),” Essex, UK: Pearson Education, pp. 1–808, 2016.
3. [3]M. Aniche, Effective Software Testing: A developer’s guide. Manning, 2022 [Online]. Available at: https://books.google.com.br/books?id=U4BlEAAAQBAJ
4. [4]D. Thomas and A. Hunt, The Pragmatic Programmer: your journey to mastery. Addison-Wesley Professional, 2019.