Under Test

Today I was thinking about the phrase "under test" as in "we've got to get this code under test so we can fix it."


I started of thinking about the way the app we were discussing was laid out. There was (at rock bottom) some persistence stuff. Above that was (on one side) reporting and query, and on the other was some business objects. Above the business objects are some model-like objects (sometimes) and above those are UI-related code.


Starting at the "top", with UI presentation, we can test through a browser. That makes it the territory of human by-hand testing and also of browser emulators, browser drivers, and screen-scrapers. Such tests are going to exercise a fully-integrated system, and can be slow due to many features such as database access, file system access, network access to services, etc. Human testing is particularly slow, as it involves reading time, typing time, and time to evaluate results. Human tests we can measure in minutes per test. The automated tests are much faster, but we might be able to measure single-threaded testing in dozens per minute. There is a practical limit to how many of these slow tests we can tolerate. Human testing is most expensive and slow, so hundreds of thousands of tests per week is not reasonable, let alone per day. Human testing can be reserved for the areas requiring special, creative, intelligent effort.


Under the UI layer are code-behind classes, which are testable via normal unit testing techniques. The tests won't be very interesting, but mocking and dependency injection can make them crazy fast. We could have many thousands of these trivial tests, and they could run in tiny fractions of a second each. We could tolerate a lot of tests here without feeling like our development is dragged down. We can afford the tests to be plentiful and trivial.


Classes that show good cohesion will have some kind of Model class under them, allowing for an alternative UI like fitnesse or cucumber. The AT framework will take up some time in overhead, but hundreds of test steps per second are still possible, and tests can relatively straightforward to write. We can make many tests and tolerate hundreds of them before it becomes too expensive and they are pushed offline to the CI system. Badly written tests can be hard to write and very slow to write, and are not uncommon. These hasten the day when running "all the tests" is too frustrating for developers.


Using mocking, we can do a lot of tests on the Model classes, crazy fast. Model layer tests might be some of the more important tests we run, as they ensure that the code behind the interfaces is sound. This is a great place for us to load up on fast unit tests. We should measure their execution in thousands per second.


Under Models are the business objects, which can be tested with normal unit testing techniques, and many thousands of tests can run in just a few seconds (if we can keep the database out of it). This is an area where we need a lot of tests, and can easily afford to run them all many times per day.


Queries and reports are a different story. We can write tests for them, but we can't tolerate the cost and expense of sitting around waiting for them. Reports are particularly rough, as there are going to be some onerous setup requirements for those. We're probably going to run them in CI and also maybe just once before pushing code to the CI system if we are diligent workers and think we might have affected them.


So what's the point? With this in mind, consider what "under test" means. Testing is arranged in layers from the top of the system to the bottom. Code in the UI is up high, not "under" many tests. It may not be tested very well at all, depending on how thorough the UI tests are and which ones the human testers choose to run. It's sketchy.


So if a system has code making decisions in the HTML templating system, that code is barely under test at all and may never be safely under test. Changes there will always be dangerous. Changes to the code-behind only slightly less so. If there is a model layer, though, it starts to look pretty healthy. The business objects, are under their own unit tests, the model layer's tests, and the UI tests. They have a thick blanket of protection, and can be refactored with relative impunity.


Get it? "Under Test" is all about confidence to make changes. Code in higher layers is under fewer tests, so code needs to be extracted and moved downward, downward, downward until it is just high enough above the persistence layer to be tested quickly and well. Get it "under the tests."