Monday, October 11, 2021

The inefficiency of tests

So a given web application has an architecture that involves a UI and an API and under that some domain objects, data, etc.

When a new feature comes up, there is a gherkin test that does NOT go through selenium, but directly to the API. In developing the gherkin test, the team drives out the changes needed by the API and gets it working.

The gherkin test is a "story test" and checks to see if the system (behind the UI) works correctly. It does data, persistence, etc in a safe way.

But to build the code, you are doing TDD directly on the API and the deeper domain objects. As you do, you are refactoring and committing (of course).

The microtests and the gherkin tests together are super-fast, so you run both in the auto tester. The auto tester re-runs the tests any time the code is changed and compile-able. This means the tests are run many sometimes more than once a minute. You're always clear where you stand.

But of course, there is a web page to deal with. You create render tests that will turn data (specifically the kind needed or returned by the API) into a web page. 

Now, these are back-end rendered old-school style. You get that working correctly.

It works, but is it ugly? Is it awful? Sigh.

Okay, you fire up the webserver and look at the page, and sure it's ugly. You work on the CSS and make it look better. You're using a development web server that reloads the page when the HTML or CSS changes so fixing the aesthetics and flow is fairly quick (if tedious).

But where is your automated test to be sure the wiring between the web page and the API is working correctly? 

Okay, you write some selenium tests. Just the necessary ones, maybe just automating tests you were doing by hand to check out the web page(s) rendering(s).

It just so happens that someone built a testing tool that pretends to be the webserver! You write some HTTP-level tests that do the GET, POST, PUT, DELETE and they run quite fast and give you confidence. You might eventually replace the selenium tests with HTTP-level tests but for now you leave some of them in place.

Hey, the app works! You deploy

You're ready to move on to the next feature, but your friend brings up a problem. 

Your friend mentions that some of the code exercised and checked by the UI test is also checked by other tests.

The render tests check the DOM, the API tests check message-passing, the microtests check underlying functions, the UI tests are end-to-end: while there are many different tests, many code paths are indeed covered by integration tests and end-to-end tests as well.

Your friend says "this is inefficient because the same code is getting coverage more than once."

You smile.

You know it's not inefficient, because you aren't measuring efficiency by total lines written or the number of times you had to run the tests, or by some standard of minimal coverage.

You know that human efficiency is high - it's a quick, reliable, and safe way to create a system that can be safely refactored, extended, modified.

You value safety over efficiency, and efficiency is a side-effect.

Friday, September 10, 2021

FIRST: an idea that ran away from home

Quite some years ago, Brett Schuchert and I invented the acronym FIRST for micro-tests, though we called them "unit tests" (as was common at the time).

It’s grown a following and has been widely repeated. At this point, the idea just exists in the aether, and authorship isn’t often considered or cited. I suppose it has become "common knowledge" at least in some circles. It's usually not even given a citation, so I guess it's become a thing in its own right.

I’m still proud of Brett’s work and my small contribution to it. I'm glad it has taken on a life of its own, but I'm aware of when it's poorly described or when it's corrupted and am offended when people present it as their own unique work (or take praise for it, knowing that it is not original).

I get it, though. I'm sure there are many people whose work I didn't know how to credit, and whose work I may have likewise twisted to my own ends or interpreted in my own context whether I realized it or not. 

The acronym is pretty simple

  • FAST - you can run all your microtests so quickly that you never have to decide to run them later or now.
  • Isolated - tests don't rely upon either other in any way, including indirectly. Each test isolates one failure mode only. 
  • Repeatable - The test always gets the same result. This is largely a matter of good test setup and cleanup, but also the consideration of things like time of day, network status, database, global space, configuration, etc never being changed. 
  • Self-validating - a test is pass/fail. You never have to examine the input, output, or system state to determine if a test has passed. It is either green (passed), red (failed), or else the test did not run to completion (error), and it states the fact very clearly.
  • Timely - microtests are not to be written in batches, either before or after coding. They are written with the code, preferably just before adding a line or just after finishing a tiny change to a function.
Some people change the T to Thorough because they don't want to encourage TDD. 

I think this is a mistake. It's not "just as good" to have high coverage from a batch of tests written after the code is finished. It's not even nearly as good.

Done correctly, the tests inform and guide the code. 

  • They help us to consider our code first from the point of view of a user of the code, which results in more sensible and cohesive APIs. 
  • The tests make acceptance criteria a primary concern. If we don't know what result we are expecting, we can't write the test for it.
  • They make testability a primary concern: we can't write a test if we can't call the method or can't observe the result of making the method call. There is no "untestable code" if the tests come first.
  • As we realize we have corner cases, we add more tests, so that it's clear when we have covered the corner cases in the production code.
  • Because tests are written first, they are a kind of standalone documentation - you read the tests to understand the code. When tests are written after the code, they tend to take the structure and algorithms of the written code for granted: you must read the code to understand the test.
  • Whatever code passes the tests, that code is sufficient. The tests help us recognize a complete state.
  • Since the invariants for the production code are tested automatically, we can refactor the code to a different shape and design with the confidence of knowing that all of the tests still pass so we haven't violated any of our design invariants.
  • Each time our tests pass (run green) we are invited to reexamine the production code and the tests for readability and maintainability. This allows us to practice code craft continuously.
  • Because our intentions for the code are captured in the tests, we have externalized our intentions. We can be interrupted and quickly regain our context in the application - something that can't happen if we're in the middle of transcribing a large plan from our heads into the codebase.

Done post-facto, the tests have no way to influence the organization and expression of code in any meaningful way. Writing tests after the code in order to increase coverage becomes drudgery.

Where to find it.
Any quick google search will turn out dozens of articles. 

Wednesday, August 11, 2021

Changing Axioms

 (originally posted in march 2008, republished with edits)

I read a mailing list entry in which one fellow (who? I can’t remember!) asked another:

“Do you want to get better at what you’re doing, or find a better way to get the results you want?”

I’m a sucker for a good one-liner. That one had me thinking, and as I’ve had other conversations about innovation, I keep coming back to that line.

In many Agile practices, we work really hard for a week or two, and then hold a retrospective. The purpose of the retrospective is to find ways to work more effectively for the next two weeks

As we develop better software, we also evolve a better team. We may use “tricks” such as tracking our velocity and recording blockages on our ‘waste snake’ to provide data for our decisions, and we use gut feel to evaluate those things that feel like collateral effort to us.

If the practice works, we will see incremental improvement in the team. We will develop ways of avoiding special variations, and we will learn to accept our normal variations. 

It will make us better at the way we do things now.


XP didn’t come from a series of incremental improvements to waterfall processes. 

I wasn’t there when it happened but it seems that they took on a change in axioms. They reimagined the development process.

They didn’t strengthen the contracts between groups but pulled all the decision-makers onto the same team.

They didn’t find more careful ways to preplan the code they were changing, but rather decided to lean radically on volumes of tests.

They didn’t build practices to improve their anticipatory design, they decided instead not to anticipate at all and simplify their design to allow future change. 

At the time, this was radical stuff.

I’m sure there have been many other less-successful process mutations, but there is no evolution without mutation.

The man behind the iPod, iPhone, and MacBookPro has had some less successful product ideas, too. Some exciting high-concept products didn’t make it in the wild. But then some new ideas become category killers.

How do we learn to make the axiomatic changes that lead us to radically better ways to get what we want?

Friday, August 6, 2021

Outliving The Great Variable Shortage


Originally Posted  on 2/26/2007

One of the more annoying problems in code, confounding readability and maintainability, frustrating test-writing, is that of the multidomain variable.

I suppose somebody forgot to clue me in to the Great Variable Shortage that is coming. I have seen people recycling variables to mean different things at different times in the program or different states of the containing object. I’ve witnessed magic negative values in variables that normally would contain a count (sometimes as indicators that there is no count, much as a NULL/nil/None would). I might be willing to tolerate this in programming languages where a null value is not present.

Yet I have seen some code spring from multidomain variables that made the code rather less than obvious. I think that it can be a much worse problem than tuple madness.

I have a rule that I stand by in OO and in database design, and that is that a variable should have a single, reasonable domain. It is either a flag or a counter. It is either an indicator or a measurement. It is never both, depending on the state of something else.

It is sort of like applying Curly’s law (Single Responsibility Principle) to variables. A variable should mean one thing, and one thing only. It should not mean one thing in one circumstance and carry a different value from a different domain some other time. It should not mean two things at once. It must not be both a floor polish and a dessert topping. It should mean One Thing and should mean it all of the time.

Surely I’ll take a shot from someone citing the value of a reduced footprint, and I won’t argue very long about the value of using only as much memory as you must. In C++ I was a happy advocate of bitfields. I haven’t been overly vocal about using various data-packing schemes, but I think that it can be a reasonable choice for compressing many values into a smaller space, but I will maintain that multipurpose variables are a bad idea, and will damage the readability of any body of code.

I suggest, in the case of constrained footprint where there truly is a Great Variable Shortage (GVS) that if (and that’s a big if) the author absolutely MUST repurpose variables on the fly that it is the lot of that programmer to make sure that users of the class/struct never have to know that it is being done. Never. Including those writing unit tests. The class will have to keep data-packing and variable-repurposing as a dirty secret.

Perhaps a strong statement or two in rule-form should be made here:

  1. A variable should have a SINGLE DOMAIN.
  2. One must NEVER GET CAUGHT repurposing a variable.

We don’t have to make do with the smallest number of variable names possible. Try to learn to live in plenty: use all the variables you want. For the sake of readability, consider having a single purpose for every variable not only at a given point in time but for the entire program you’re writing.

Monday, June 1, 2020

A Whole Lot Of Nope

In light of recent outrages, there are a lot of people posting bloodthirsty things on social media. If you post bloodthirsty things, I want you to know I don't stand with you on that. I may agree with your underlying cause(s) and reason(s). But I don't want people to be killed.

I don't want looters to be killed.
I don't want protesters to be killed.
I don't want suspects to be killed.
I don't want curfew-violators to be killed.
I don't want civilians to be killed by cops.
I don't want cops to be killed by civilians.
I don't want politicians killed.
I don't want community leaders to be killed.
I don't want bystanders to be killed.

I'm not claiming that all of these things are equivalent. The only equivalence is that I want all these people to go home at night, and justice to be accomplished without bloodshed. I'm not suggesting outrage is unfounded. I'm not saying nothing should be done.

I will suggest that if escalation and bloodthirst were really the answer, they probably would have worked by now.

Monday, May 4, 2020

Speech Hacks to be More Decisive

I was in a leadership program with Christoper Avery some years ago. In that program, people would say “don’t should on yourself.

Another friend had years ago told me that “should” is the saddest word in English because it means you see value in something, haven’t done it, and probably won’t.

I embarked on a quest to get rid of some aspects of self-defeating speech:

  • Instead of “I’m sorry” say “Thank you”
  • Instead of “No thanks, I can’t eat that” say “No thanks, I won’t eat that.”
  • Instead of “I don’t know how” say “I haven’t learned YET
  • Instead of “I should ” say “I may ” or  “I would like to “.

You know, these little things make a difference for me.

The speech patterns are more decisive and confident, reflect agency and choice, and generally help me avoid shame (in myself) and appearing uncertain or indecisive.

These are just little things, and it's not magic. One a person chooses an attitude they need to find language that supports that attitude.

My speech changes support the internal decisions I've made about how I want to address the world.

I am no psychologist so I can't say why it works for me or whether it will work for you. This isn't a prescription.

If you want to try some or all of these, then go to it. I didn't originate these ideas and you don't owe many any credit if it works wonders for you.  If it doesn't work for you, well, you were warned.

If you try these hacks, then please:
  • Try them for yourself and on yourself. 
  • Do not insist that people use these terms. 
  • Do not chastise or pillory other people for not using these terms.

Does changing a speech pattern really change a person's attitudes? Probably.

Are we wandering into political correctness and the euphemism treadmill right now?
Gosh, I hope not. I'm no expert on those and see both advantages and disadvantages in those things.

I don't feel the authority or invitation to lecture you on how you should/must/shouldn't/mustn't speak. I'm just offering some things you can choose to do or not.

I'm not saying you should.

Tuesday, March 17, 2020

Some Covid-19 Information

Hello all.

From time to time, I will post something that is not on-brand for Agile Otter Blog at all, but is of concern at a point in time or in celebration or mourning.

Today I'm listing some resources related to the Coronavirus COVID-19.

A quick warning: there is a thin line between being informed about the virus and being obsessed with it. You should have the facts, and these sites will help you. But don't be so obsessed that you can't think about anything else. Take precautions, follow guidelines, don't be fooled by myths, don't search for miracle home cures. Take care of yourself and your family, and try to continue living a productive and normal life.

I took time out to gather a few resources for you, but I have other things to do too. May we all get on with delivering value to each other and serving our communities of practice as well as protecting our friends and family.

Don't panic. Do take precautions as recommended.

Info and advice:
Myth Busters:

Stay healthy, and that includes not having an unhealthy morbid fascination with covid-19.