On one hundred percent code coverage

The mindset for testing for developers has changed over the years as the software industry is evolving. Even though, testing practice is not widely used, it is integrated in the development cycle of the application. As such, code coverage became a popular subject among developers and discussions have been risen in order to agree on a consensus of code coverage.

This post aims to give this discussion a push and share what I think about code coverage, what I see teams doing and what I understand that is effective or not.

Test driven development

Test driven development has been (TDD) adopted for developers in order to achieve high quality software as well as to keep it evolving over time and avoid the fear of change. Therefore, [1] and [2] describes as TDD being a three stage flow (red-green-blue), is not what I identify in the projects I have worked on.

Most of those projects were using a combination of ITL(interactive test last) or not testing at all, the test was given to a QA (Quality assurance) professional. In this scenario, the team wasn’t getting any value at the time to keep the tests up to date or even to write them. This is a entire discussion that I will not approach here, though, it is what possibly leads to miss leading metrics. As often management try to force developers to reach 100% coverage just for the sake of it, or because they saw they could use that to force some kind of behavior on developers. James Carr catalogue named this anti-pattern as “The Liar”, which Dave Farley uses as a base line to discuss the subject on his video [3] ([4]) also mention the coverage goal being a missleading understading on his video about Behaviour Driven Development).

TDD is a safe net for developers to keep improving the code, communicate intention and also a culture to follow. As Dave Farley says on his video: practice TDD and avoid the liar trap.

Quality gates

Quality gates are used to enforce some minimum rules during the software development life cycle. Among different rules we can list:

  • Code linting
  • Test suite
  • Security checks
  • Performance budget

Many would argue that code coverage would have its place, which I agree. We could use code coverage as a gate to not allow code into production, if the code base has less than X percentage we fail the release process. Therefore, besides being a quality gate, it should be a indication of the test suite health.

The team, should trust the test suite and the coverage should reflect the health of the suite. Which, in many cases this is not what happens. The common approach is to target X percent of coverage no matter what.

Avoiding wrong metrics

My experience tells that for many developers tests are a matter of needed, kind of obligation. They write the tests not because they want to be proud of the work they do, or because they want to give the next developer (that will maintain that in the future) a hint on what the code was built to support or not.

As such managers, try to enforce the idea that forcing developers to reach X percentage of coverage will increase the code base quality. [5] recorded a 3 series video going through the famous Gilded Rosa kata, which to goal is to refactor the code given. Around the minute 15:59 of the first video she depicts an issue in the tests that she had. Even though she changed a critical parte of the production code, the tests were passing. The code had one hundread pecent coverage. It was not giving the desired feedback.

Of course it was a kata and she beautifuly depicts this problem as she goes through the code. Still, I see developers making they proud of themselves because they have X percentage of coverage.

This video series alone points how useless matric of X percentage coverage is. Code coverage should be a side effect of the test suite quality [4], which software developers can rely on and understand that tests are a safe net for continuous improvement of the code.

References

  1. [1]K. Beck, TDD by example. Addison-Wesley Professional, 2000.
  2. [2]M. Fowler, “TestDrivenDevelopment,” 2005 [Online]. Available at: https://martinfowler.com/bliki/TestDrivenDevelopment.html. [Accessed: 31-May-2021]
  3. [3]D. Farley, “When Test Driven Development Goes Wrong,” 2021 [Online]. Available at: https://youtu.be/UWtEVKVPBQ0?t=243. [Accessed: 05-Jun-2021]
  4. [4]D. Farley, “BDD Explained (Behaviour Driven Development,” 2021 [Online]. Available at: https://www.youtube.com/watch?v=zYj70EsD7uI. [Accessed: 27-May-2020]
  5. [5]E. Bache, “Introducing the Gilded Rose kata and writing test cases using Approval Tests,” 2018 [Online]. Available at: https://www.youtube.com/playlist?list=PLuvRKxeqrv4K-rn0zxHPNiXOWBkP9ZZIH. [Accessed: 31-May-2021]