Wednesday, December 31, 2008

Quality Rant

Men more brilliant than ourselves have tried for decades to get the idea of quality across to businesspeople and tradespeople (including software craftsmen) and have had only very limited
success.

TDD is just another grandchild of Quality.
http://en.wikipedia.org/wiki/W._Edwards_Deming#Deming.27s_14_points

Deming's points didn't stop applying just because we're in software. The excuse,"Our problems are different" was specifically listed as an obstacle to real improvement.

But still we feel have to justify the desire to increase quality of our products. Does this seem silly? I spend far too much time trying to get people to build quality in via TDD and JIT inspection (AKA pairing) and collaboration, but still they feel that this is slowing them down.

"We all *know* the sun circles the earth, because it rises in the east and sets in the west. Stupid heliocentric theory is a fun pastime for intellectuals, but doesn't work in the real world."

My vent for the day, from very real frustrations.

For the lazy, busy, or browser-impaired:

* Create constancy of purpose toward improvement
* Adopt the new philosophy
* Cease dependence on inspection
* Move toward a single supplier for any one item
* Improve constantly and forever
* Institute training on the job
* Institute leadership
* Drive out fear
* Break down barriers between departments
* Eliminate slogans
* Eliminate management by objective
* Remove barriers to pride of workmanship
* Institute education and self-improvement
* The transformation is everyone's job.

Monday, December 29, 2008

Put simply

What I believe about software development in simple terms:

If your team will not pair AND test, if your Customer will not prioritize, if your customer and developers will not collaborate, then nothing you are doing process-wise matters even a little
bit.

If you have these problems, you might want to forget your standups and scrum-of-scrums, scratch your code reviews, disband the QA team, ignore any promises about delivery. Better to spend your time collecting resume fodder.

If these are your problems then change your company, or change your company.

Saturday, December 13, 2008

Opportunity Cost & Priority

If you have a fixed number of resources, then the only way the team can take on a large new story is by not doing an equivalent amount of previously-assigned work. If the team is honestly working as well as they can, then even small tasks cannot be taken on without other tasks slipping. If they are lazy so-an-sos, or have been withholding effort from you then they can absorb more work.

So why is is that when a team can absorb more work without letting anything slip they get rewarded, and when they cannot they are criticized? Doesn't it seem like they should be given credit for really working at capacity all the time?

Or maybe it's a wise leader who quietly keeps a small reserve so that he can respond quickly to new requests. Should I, as a manager, keep one or two people working on easily-interruptable, low-value activities so that I can respond when the emergency of the day comes along?

There are clearly advantages to both. I've been running fully-committed for a while and there is almost always some kind of emergency or special project that needs tending at some point in the week. It is always "very important." I suppose a wiser fellow would keep some developers in reserve.

OTOH I would not be able to look anyone in the eye and say we were 100% engaged if we weren't. I could phrase it cleverly as "I can't afford to put any more man power into this", but that is a cop-out. Say I hold my reserve and nothing happens -- then I actually prevented the team from delivering value, which seems obscene. Or if I hold the reserve, and something comes up; won't it seem odd that there were no available horses for task A but there are for task B? Clever phrasing is not truth. I'm just not a "convenient distortion" kind of guy. I value a clean conscience. I want to work in a transparent shop, so I have to maintain transparency.

I will keep people fully engaged in the most sustainable way possible, and be sure that each new "hot case" or "emergency" has a known opportunity cost. When someone says "this has to be done by Tuesday" then I must always follow with "Okay, which of these equal-or-greater sized tasks can afford to slip?" If nothing can slip, then clearly the case is not so hot or the emergency not so urgent. Maybe the stake holders can fight over it long enough for the developers to free up.

I am not going to sacrifice integrity here. I guess that means I should be ready to deal with the criticism when I have to push back for someone to make a priority decision. Nobody said a clean conscience was without cost.

Thursday, December 11, 2008

All Together Now!

Today I was in a meeting that made me smile. When I came to this team four months ago (Aug 4th) and took up the agile coach role, I talked about changing how they did ATs. In line with conventional Agile thinking, I told them that the Customer should write them with the aid of QA as subject matter experts. Customer should own the AT, but the QA group could help them flesh out the tests so that they serve the team as requirements and regression protection. At the time, it simply didn't happen.

The QA group were writing the ATs on their own back then. This was strange, because they didn't run the ATs and certainly didn't trust the ATs, but they were writing them nonetheless. And they were disliking it. Sometimes the developers wrote the tests for QA, or in pathological cases rewrote them quietly. I am experienced enough to give a consultant's "okay" (meaning "if you insist") and move on to other practices.

We finally moved AT responsibility to the product group, and few of them put the real effort into learning to write them. The practice grew slowly until it became clear that having ATs in hand made it quicker to develop features. A few of the product people really stepped up then, but QA was no longer involved at all.

Our QA manager moved QA from tail-end release candidate testing to the working with active feature development. Errors are being caught sooner and fixed sooner than ever before. It's new to them, though, and new behaviors always require some settling in.

At the meeting today the "testers" were a bit frustrated. The story card just didn't give enough detail (nor should it have). They were told that the story card was just a token and not a document. They were unhappy with that. Then they were told to use the AT as the basis. At this meeting they were unhappy again because they found too many situations not covered in the AT.

I was delighted! This is exactly the collaboration I was looking for! Now the QA group has an active role in defining the tests for features being developed. Rather than a veto, they have a voice. The collaborations in our group have never been richer. The product group is driving the process, the QA process is actually establishing quality rather than proving its lack, and the development time is able to develop higher-quality work

We're not in perfect shape yet. There is a lot of technical work and mentoring before we can claim to be truly doing TDD, and we don't have all the tracking and control we'd like, but we're moving forward quite nicely.

I am looking forward hopefully. The transition is looking better than ever before.

Raspberry Jam

The law of Raspberry Jam is "the more you spread it around, the thinner it gets." A useful principle when dealing with work-in-progress, attention span, goal-setting, etc.

I found this gem recited in an email a while back and saved it, without really remembering who wrote it. If you know, please feel free to help attribute it. It quotes Goldratt's /Critical Chain/:

There’s a lovely example of the evils of multitasking in Goldratt’s Critical Chain.

Suppose you have three things to deliver: A, B and C. Suppose they all take, I don’t know, 2 days each to complete. If you did them one after the other, A will be finished by day 3, B by day 5, and C by day 7. AABBCC.

Now suppose you want to satisfy the people who commissioned B and C that you’re paying attention to them, and you decide to multitask. So you decide to split things up, and work on them in one day blocks, ABCABC.

The result is that now A is delivered on day 5, two days later than before, and B is delivered on day 6, one day late, and C is delivered on day 7, no earlier than it was before. You’ve potentially pissed off two people for no benefit.

If you work through the example with shorter and shorter timeslices you can see that the more finegrained your multitasking, the later all the tasks become.

Of course it’s worse in the real world because of the overhead of context switching, something computer scientists know all about.

I find that when I allow myself or my team to be overwhelmed with tasks, we lose all effectiveness to get any of them done. I have vowed to never interrupt a task in progress other than if 1) I don't want the task to be completed at all, or 2) ordered to do so by a superior. I think that law is easy to work with, and it helps a bit. It is hard to defend, because people want everything imaginable to be done or at least in-progress. But that's not lean, not agile, and not even reasonable.

Make value, not inventory. Don't spread the workers' attention thin.

Saturday, December 6, 2008

Save Your Project

Good advice on how to keep your project from circling the drain.

Items #7 through #11 simply do not get enough play. They are:
  • Prioritize
  • Make features, not inventory
  • Leave the team alone.
  • Streamline decisions.
  • Build quality in.
When I see teams in trouble, the hardest things to teach are these. Prioritizing means that you have to declare something more important (or at least more urgent) than another. People simply don't like to do that.

Making features (not inventory) means that we need to "see the whole" and measure our time from specification to *delivery* not just coding time. We can easily optimize the wrong things and create nothing but trouble, heartache, and last-minute drama.

The answer to getting things done is not to have more things in progress. Start rate has to match up with completion rate. If you start too few things, developers are idle and money is wasted. If you start too many things, then it is harder to get any one thing done. You end up with 80% of the work 80% done and nothing to ship. The smart (lean) thing to do is measure your completion rate, and feed in exactly the amount of work that can be finished. Adding more work doesn't help move things along by creating pressure, it slows things down by creating a large inventory of partially-done work.

Because your capacity is finite, you must prioritize. If you cannot prioritize, you clog the system and nothing gets done. You can try to increase your team's capacity to produce, but that is a matter of streamlining, staffing, training, and tooling and not a matter of stomping, screaming, cheerleading, or cramming more work into the funnel.

Leaving the team alone is a hard sale. It's absolutely right, but everyone suddenly decides that they could manage your team better than you can if they could directly handle your developers. They're flat-out wrong, but they are very well-intentioned and convinced. My manager and I have over 60 years of experience between us, but there are a million low-mileage would-be managers who think they can do it better. We struggle to keep our hands off too, because sometimes we want to dictate. I struggle more, as the junior of the two with only 30 years in the business. He's wiser. Luckily I listen to the developers I serve, and that keeps me from too much trouble.

Streamlining decision-making gets more difficult, too. As pressure mounts, people want to help with decision-making (and decision-reversing). It's just a bad idea. A team that is moderately productive can certainly go to hell in a hand basket once they get multiple work queues, multiple priorities, and increased thrash.

Quality may be the hardest sell. It is counter-intuitive that test-driving, testing more, thinking more carefully through code ramifications, and programming in pairs will actually make the team *faster*. Again, it's because people want to measure programmers' typing speed instead of how long it takes to successfully release a product increment.

See what the other author has to say, take a look at Lean Software Development, and join me back here for some followup later.

Thursday, December 4, 2008

Trickle Effect and Project Publicity

I have to be a little worried about the trickle effect, as it was described to me by my Andrew Cohen (boss and mentor) today. Since moving to Agile, the group produces small increments of functionality every single week. The Customer has more opportunities to decide that a feature is "good enough for now" and move on to more urgent work.

That sounds like a good situation. Sure the team is as responsive as it can be to market needs, and sure the most important work is what gets done, and sure it's in measured weekly bits. But this isn't what people come to expect from a software team.

Since there are no large quarterly or monthly releases, it seems like nothing big ever gets done.
If we release every month instead of every quarter, we get 1/3 sized releases. If we release every 2 weeks instead of every 6 months, we get 1/13th sized releases. They're "less" though "more often."

Where is the PR event, the release party? Is last week the point of reference from which productivity is measured? Does it seem underwhelming that we do 1/4 as much in a week as we did in a month? Do we forget the sample size and see only the functionality delta shrinking? The increasing smoothness and predictability of the team works against it.

Releasing less often would stockpile enough work to be appreciated, but would delay value to customers. It would be a PR success but at the expense of the people the team exists to serve. Maybe this is the problem inherent with trying to please spectators and commentators in addition to customers. Withholding a finished feature feels obscene. Releasing sooner seems
better.

Maybe it would help to have some kind of publicist to report progress since the last month, quarter, half-year, and/or year. There must be some way to create some degree of appreciation for the real progress the team is making. We're instituting big visible burn-ups for new features. That should help some.

Suggestions are welcome.

Tuesday, December 2, 2008

The "D" Word

The agile value of the day is Discipline, the D word. With it we avoid being overwhelmed, snarled, and demotivated. Without it, nothing goes well. Here are my top 8 items in the "fruit of the agile spirit":
  • Refuse to complicate your system.
  • Don't break ranks
  • Be transparent even in difficulty
  • Work in very small steps
  • Maintain/increase quality
  • Impose simplicity on software.
  • Build a better "next month"
  • Go home before you break something.
These very simple things can be very difficult, but they are very important and simple nonetheless. When I find myself going wrong, it is because I've broken from these simple ideas.

Monday, December 1, 2008

Where we are.

The new team is great. We've put some extreme measures in place, but we're getting some focus on getting real completion.

To deal with the huge QA backlog, we have devoted 2/3 of our programming staff to dealing with making software releases (dealing with old bugs, bringing code forward from the dark ages to current trunk, etc). We rotate the team roster randomly so nobody is stuck doing the dirty work all of the time. Nobody expects this team structure to last. It has changed how work is done, though. More people know more things about more parts of the code, and work no longer queues for individuals. We make weekly releases.

We're not fully agile, but we've been moving the right direction. There isn't full test-driven development and full pairing, so we're slower and less certain than we should be right now. In fact, we're just hitting that hurdle where there are so many non-unit tests that it takes a half-hour to run them all. Clearly we have to refactor for speed so we can do more testing, better and faster. Tests are not uniformly the first thing people read or write. This is something we have to start pushing harder.

Pairing is starting, but it's really more like parallel programming and basic collaboration. We're not seeing ping-pong or baseball programming games. A lot of people still prefer working out the code for themselves. I figure that's partly because they're still a bit nervous about how much they know versus how much they think that they should. It will change with the random rotation. I would push harder, but there is steady progress being made here now.

The hardest part is getting used to "Done Done" and the customer driving. We're making progress there also, but it's not smooth and easy.

As for me, my biggest problem is learning to keep up on the mundane tracking. My memory is too poor, and I need to keep solid records as a manager of a team, even one that rotates regularly. I'm doing a lot of the mundane work for the team (babysitting subversion branches, etc) but not a lot of "real he-man coding" here. I've forged some decent working relationships, and try to keep on top of things.

My current fun is trying to get the tests automated and reporting on them, and mining some data from svn to build some theories there. I'm starting to learn javascript because my colleagues and some friends have inspired me to do so.

Unit Test Ice Breakers

So you're looking at this pre-existing code. You aren't sure what to do with it, but you need to get some unit tests going. What do you do when you get stuck.

  • Start by testing you can instantiating the class.
  • Test that you can call the function at all.
  • Pick the easiest bit of functionality.
  • Write an empty function name and assertion first.
  • Write the function call as you would like it to appear.
  • Refactor the existing tests.
  • Pick the most interesting bit of functionality.
  • Switch partners.
  • Read the code for obvious flaws.
  • Rerun the AT
Again, this is part of the work being done by myself and Jeff Langr.

Doing Python Wrong

Mike Pirnat found a snippet of python code. I can only suppose it was written as a joke. Python is such a readable language left in something approaching a "natural" state. This code looks like an entry in the old C obfuscated code contest. It uses a lot of the same tricks they would use back then, though we don't have any way of writing proper macros. It could have gotten much worse if they really wanted to stretch with lambdas, comps, generators, localizing methods, misusing decorators, etc.

Personally, I think it's a great example of how important naming and tidiness are. If you can make a great language like python look like a load of manure, then imagine how "clever" programmers can do evil in languages like C++, Java, or even Ruby.

Bad code is a matter of intent (or default, in some cases). Good code is intentional. All our code should not be joke code.