Saturday, April 5, 2008

I suggest we stop using Unit Testing

I suggest we stop using the term 'Unit Testing' and use the term testing instead.
The problem with the term 'Unit Testing' is it refers to a specific type of testing. To many people, it means to test a unit of a program. The smallest discrete piece that you can. In the Java world, that usually means that you have Unit Test class files that test a Java class file, and in each unit test class you'd test all of the methods in the class.
There are some who would create unit tests for each method and call the testing effort done. Some tests are better than none, but it's irresponsible and foolish to stop automated testing at this level. Unit tests are far too robust to catch the nuanced causes that can make a program function different than the intent of its creators.
First, let's get rid of Unit Testing as a term. It's already loaded and it means many things to different people. I think this type of testing is better described as isolation testing or micro testing. My preference is to have a set of method level tests that exercise the intent of those methods.
I prefer the term Isolation Testing because the tests are testing the elements of an application in isolation from their dependencies. In isolation testing, it's preferable to have tests that run on mocked data and stubbed services. They're only concerned with the isolated elements.
Beyond Isolation testing, I prefer to also have higher level integration and situational tests. Integration tests exercise how classes work together. They are more fragile than unit tests, but I would argue that they are more important. They will break earlier than unit tests. If it is feasible, use real data for these, mocking or nuke and pave is ok if situational tests are also in the suite.
Situational tests are a full end to end tests that outline scenarios and user use cases. I like using tools like Selenium to capture scenarios from the user's perspective with tons of assertions. If it's at all possible, I prefer that real data be used for these tests. The more fragile tests you have, the better.
The final automated testing piece is defect regression tests. For each defect that is reported against an application, one should try to create tests to replicate how the software fails and test against it happening again. Some Selenium tests could be very useful in doing this, but tests at the integration and isolation level are good too.
One thing to note with the fragile tests: they will break early. It is very important to have an automated build server like Cruisecontrol or Hudson running automated build scripts after each modification to the source repository. The role of the automated build server is to test the code and let the team know when it has failed. It is important that the continuous build server runs a set of tests that completes in a reasonable period of time for checkins.
For one project I had the server run tests that would take 5 minutes to run. It would run 5 minutes after a checkin. 10 minutes after a checkin, the team would get notified if a submission passed or not. This was helpful at the end of the day. A responsible person could check in their last submission of the day and wait 10 minutes to see if the smoke test passed.
The last component of my testing strategy is to include static code analytics and code coverage tools. I like Cobertura and Crap4j. Findbugs is also good. The role of static code analysis is to measure the complexity of the code and the amount of coverage in the unit tests. The more coverage, the better. The less complexity, the better.
The above is my proposal to replace the term Unit Testing and the practices that we've come to know as unit testing in software development. If it is at all possible, I would encourage developers to try and write tests first also.

No comments: