Tuesday, July 17, 2018

Q&A on Velocity, Part I


Having had many conversations about velocity (and many blog posts here and at the Industrial Logic Blog) one day I decided to write up an example of the conversation as if it were happening between two characters named A and B.

The first question asked by A was the first question about story points that a younger agile otter asked in his first XP project.

Most of the questions were asked by my younger self at some point or asked by someone I have worked with.

By the same token, the answers came from myself and people I know too. I've internalized these conversations to the point that I'm not sure what questions were mine and which were asked of me, and which answers were original and which came from mentors (managers, coaches, developers, etc).

Suffice it to say that I'm represented (at various points in my journey) by both the questions and the answers. As such, the point of the dialog between A and B is not to humiliate or pillory either A or B. The questions are pretty reasonable (most of the time) and the answers are not only honest but reveal deeper truths about software development and agile methods -- most of which one cannot acquire except through years of experience or mentoring.

At no time is it to be assumed by my publication of this series that I recommend using story points or velocity. I do not.  Even so, if people are using them, they should understand them at least until they realize that they are better off without.

So, here is part 1, annotated.

A: How long is a story point in real time?
B: How many did you do in your last sprint?
A: 23
B: Then, for you, it's a 23rd of a sprint
A: But the time before that it was 19
B: At the time it was 1/19th of a sprint.
A: But how long is it really?
B: We've determined that.
This was a major breakthrough for me early on. I didn't realize that roughly the same amount of work is done every week. Story points are a rough sizing of work, not a measurement.

I was once informed that every estimate involves not only effort but risk and learning. Risk and learning are often the larger part of each estimate. So when someone assigns a larger estimate, it is usually because they don't know how much they're going to have to learn to do the work or because they don't know what the risk is of it having wider unwanted effects on the schedule.

Many developers and managers refer to stories "blowing up" -- where it was estimated as a 2, but in reality, it was more like an 8 or 13 or 21. It's hard to know where the risks and learning will occur.

Sometimes a story assigned an 8 or 13 turns out to be easier and less risky than imagined, and the story is done in the time it would take to do one that would have been assigned a 2.

This doesn't mean that the team is doing a bad job, it really reflects more on the variability of the work items instead of the lack of consistency of the team

When the team is not familiar with the source code of the project,  it is more common for greater numbers to be assigned and it for the stories to "blow up" or have defects. That's what happens when you don't know your way around.

But either way, the size of a story point isn't very meaningful -- it's a guess to begin with, and its "real size" is not only knowable in arrears, but is also variable. It is a unitless unit of measure, a bit of nonsense that people sometimes find useful for capacity planning.

It's almost always misused. An option is to stop using story points.



Go to Part II


Thursday, July 5, 2018

The Scatter-Gather Method of Software Development

There is a certain theory of agile development which is built on factory thinking, using division and management of labor to accomplish large projects.

The Plan


You have a big job to do?

Okay.
Scatter:

  • Divide the project into pieces
  • Resource the project (acquire the right number of "teams")
  • Assign pieces of the project to "teams" 
  • In the "teams", a lead character will divide the pieces into smaller and smaller jobs
  • The small jobs will be assigned to individuals who work for the lead

Gather:

  • Individuals complete their small jobs
  • The small jobs are integrated/combined/summed by the lead
  • When small jobs are done, the lead delivers the finished work of the "team"
  • "Teams" who completed the work are released
  • The project office then combines the parts
  • The integrated, completed project is delivered

Please note that there is an inverse pattern here: "Teams" are gathered during the scatter part of the process, and scattered during the gather part. But let's not focus on that right away.

Why do you put quotes on "team"?

A team is a group of people aligned on common goals, who work together to achieve those goals. Such a team works together every day, and each member works for the best team outcome. That is a team without quotes.

There is another use of the word, though. In many organizations, all the workers are individual contributors who are encouraged to get their own work done. 

They might occasionally offer advice or be a "second pair of eyes" (cf Rubber Ducking) for another person, but the only real connection between them is that they have the same boss. 

"Teams" (quotes included here) are an org chart conceit. 

I find that scatter-gather style work requires exquisite planning and estimation, and for one person to be tending to another person's assigned work (remember it's assigned to individuals) puts the individual's (aggressive) schedule at risk. 

Each person absolutely must complete their own piece of the work on time, else the whole schedule cascades into failure.

Quite often, the manager/lead is posted to act as a "blocking troop" to punish any person who falls behind. 

Often leads are chosen for their ability to operate as an individual contributor. 

Sometimes leads are chosen for their exquisite planning skill. 

Occasionally, a lead becomes a lead because they have a penchant for working as blocking troops. 

In every case, it is a token of respect to be "raised" to the level of a team lead. Different organizations and sub-organizations just respect and promote different qualities.

Integration

In theory, when all the individuals finish their parts, then those can be integrated and tested, and the parts passed up the tree for integration and testing at a higher level, and then finally the organization will resolve all the integration problems and mismatches and release the software.

Ideally, there will be no integration problems because of the upper-level project team's exquisitely detailed design and planning. People are often promoted to this level because of past success and because of their talent and passion for planning. The goal is to have everything worked out up-front so that integration problems do not happen.

But they do happen. They almost always happen. They're happening right now.

There are always some misunderstandings, some kind of design drift, some platform issue that had to be worked around and of course some of the small jobs took longer than anyone expected. Some late parts had to be left out (temporarily) or revised during development (thus damaging the validity of the grand plan).

How do scatter/gather organizations resolve those problems? Generally speaking, the people at the project level don't have the technical skills to resolve these problems.

Luckily, even though they tend to run as project organizations (a convenient but prevalent organizational conceit) "teams" tend to be composed of full-time employees working long-term on projects that tend to be related to a single code base. Therefore, the members from the original "team" may still be present, and may still be on the same "team."

Integration problems are addressed, then, by temporarily borrowing-back the members from the original "team" who can diagnose and resolve issues.  Once the problems are resolved and the system is sufficiently tested, then the original "project" (really a product increment) can be "released" to customers.

Remember that integration problems are treated as exceptions.

That integration problems always happen in scatter-gather is only because the planning and detail were not exquisite enough to avoid the failure, and scatter-gather is reliant on predictability and exquisite scheduling. Without sufficient up-front planning, the project will not come in on time with full scope completed.

And of course a lot of projects don't come in on time with full scope (which suggests that a more agile approach is appropriate), but the scatter-gather system is so reasonable on the face of it that many people think it the only system that could possibly work.  Sometimes people say "at scale" or "at enterprise scale." The goal of "scale" in this sense is to produce a product using several hundred or thousand programmers simultaneously and keeping them all busy.

Some organizations have made inevitable late integration problems part of their standard process by adding hardening sprints. "If you can't fix it," we are told, "feature it."

Busy Enough 

One of the problems with working at this scale is that programmers are not cheap. Even the cheap ones are not cheap. There is a constant "burn rate." An hour of time costs the same whether the teams are productive or idle. And it's a VERY LARGE NUMBER.

If you are paying, and it's a VERY LARGE NUMBER per hour, then it's important that all the people ("resources") are kept productive at all times. This isn't snark, or hyperbole, or foolishness. It's a VERY LARGE NUMBER and you need to be sure that you're getting value for the cost of that hour.

So when teams are released from one project, it's keen to have them starting another ASAP. This is a tricky bit of "scheduling tetris", but many managers above the project manager level have significant skill at scheduling. Sometimes, to make the scheduling work, they have to shuffle teams in or out of a project where the team "members" have no familiarity with the code base or domain, but we know that given time a developer (QA included) can develop skills in any code base or domain. It's a matter of good communication and management, which the "leads" should be able to handle or they shouldn't be leads.

When I first began to manage teams, it was in a scatter/gather organization. I was told that the first rule of management is to always make sure all the team members have enough work to do.

I was given the VERY LARGE NUMBER which is the hourly burn rate, and reminded that an idle team costs the company many many digits of loss per minute.

Oddly, I have no idea how much value or income the team's product produced per hour, day, month or year or what the ratio of cost:value was for the project. I was instructed to focus on the costs and get the maximum quantity of work possible to justify the burn rate.

I've heard this First Law repeated many times. It is a corporate mania that the profitability of the organization will plummet at a rate of $VERYLARGE/HOUR if employees are not busy enough. It could destroy the company (and certainly would, if the burn rate exceeded the rate of income for even a relatively short period of time; days, weeks, months. It's inevitable).

This First Law is one of the problems with the hardening sprint. The staff in a hardening sprint end up waiting for something to be broken. Idle resources are expensive, and people must be kept busy, so hardening sprint staff tend to be part of some other scatter/gather project and likely on the critical path.

In other words, the hardening sprint is usually another organizational conceit.

Some Flaws 

Since everyone must be busy enough, by the time an integration problem is visible (at the end of the "gather" process) most of the people who worked on the code have moved to other projects in order to satisfy scheduling Tetris.

Also, as a result of "schedule Tetris" the project code is written partly by people who didn't know the code base, or the domain, or the development stack. Because they needed to be productive and to stay busy, they did the best they could at the time. Naturally, there are some defects, integration faults, inconsistency, and schedule slips.

Errors are the natural result of putting unfamiliar hands in complicated code.

The people who can fix the problems are the people familiar with the system as a whole, and those familiar with the code that was written. Of course, they're not on the "project" anymore, and may not even be familiar with the code they wrote anymore. They've moved on.

To get the problem fixed, some of them will need to be pulled out of their new, exquisitely planned and scheduled project. They will be brought back for some time (period unknown) to the suffering complete-yet-non-working-so-unreleased project.

This puts the new project at risk, which runs the risk of cascading schedule failure.

Often the risk to the project schedule is such that more teams are brought to bear (also often unskilled in the development technology, unfamiliar with the code, and new to the domain) to try to make up for lost time. These teams are likely to introduce similar failures in the new project, adding to the integration risk.

And the increased integration risk pulls more resources into the hardening sprint, putting another project at risk of cascading schedule failure, and a combination of scope reduction and staff increase is brought to bear again.

Either a point of equilibrium is reached, projects are canceled to relieve the pressure, or some other form of hilarity ensues.

Faith

Projects are late, schedules have cascading failures, "team" members are unhappy because they can't do their best work rushing around in the unfamiliar code, everyone is frustrated that "teams" can't avoid making errors and "team" members don't help each other to learn to work better.

Managers are frustrated with descoping work to try to get projects back on schedule, with the frustration of trying to keep up on the status of each "team" member, with never knowing if the system will be delivered on time, with the nagging sense that the team is missing something or the plan is wrong in some way that will cascade into failures.

But still. Still. Even in the face of years of struggle and repeated near-catastrophes and occasional failures, they are convinced that the system is entirely workable and the problem is either:
  1. they couldn't plan and design well enough under the circumstances
  2. circumstances arose which shouldn't have, and couldn't have been predicted
  3. the plan wasn't followed well enough this time and next time should be better
  4. our people just aren't good enough
Scatter/gather seems to be the only system that could conceivably work (at this scale). 

It's a tremendous act of faith, and the organization doubles-down on faith in the face of repeated failures. Scatter/gather must be made to work. We just have to do it better.

Alternatives



Of course, much of the world has developed mature practices which use continuous integration to surface problems early, collaboration to resolve problems quickly, and using evolutionary design so that the product is runnable on day 1 and every other day.

A lot of people don't even use projects to build product increments. They add features one-at-a-time or one-"epic"-at-a-time. They build release candidates every day throughout the project. Some of them actually go live with their product increments several (to hundreds) of times a day.

I know it sounds like heresy against the One True Way, but it's still so.

I know it doesn't sound realistic. How in the world could you do this in a scatter/gather organization? Without screwing up the schedule Tetris?

And to be fair, you couldn't.

You'd have to give up the faith. You would have to give up your core identity -- you wouldn't be a scatter/gather-based scheduling-Tetris organization.

You might even have to admit that the old system that you spent (VERY_LARGE_NUMBER/hour X for N years) on is not very workable.

There are often stiff penalties for being the first one to speak heresy against the Current Corporate Way. Worst than if you'd violated the First Law. Worse than if you failed to deliver. You could actually lose your position in the hierarchy, which is the Primary Topic in a large organization.

Still, a lot of companies work alternative ways and have for decades.

Keeping the Faith

But if you have great collective faith in scatter/gather as the only possible answer, alternatives sound ludicrous and couldn't possibly work in the real world.  Anyone who thinks that they could build a working system at scale without a scatter/gather plan must have lost touch with reality -- lost the faith -- possibly their whole way of working is a peyote-inspired fantasy.

Denial is the first step in avoiding responsibility, and humanity has found a lot of success in denying facts for decades in order to maintain their conceits about how the world works. I can recommend this strategy. It keeps you from having to change right now, and if you discredit heretics then you might not have to hear from them again.

I recommend it if and only if your goal is to continue, undisturbed, in your current beliefs.

And you have supporters in the industry, so that's nice.

The true believer in scatter/gather projects, schedule Tetris, and other organizational conceits can find people in the world who are happy to sell you agile frameworks entirely compatible with your way of working.

They will also gladly rent you consultants to put the processes into place and certify you to be practitioners of the True New Way of scaling.

They'll even give you the prestigious "Agile" title to adorn your press releases and recruiting posters.

But maybe, just maybe,  if you've suffered enough for your faith and find yourself filled with doubts, you might consider some of the alternatives.

Wednesday, June 27, 2018

Peace and Curiosity against Righteous Indignation

All anger presents as righteous indignation. Have you ever been angry and not convinced you were right to be so?

It is only in retrospect that one realizes that one was in error, with embarrassment and shame.
Maybe being angry makes us too sure, too certain of our view of others, too unquestioning of the humanity of others.

Sadly, righteous indignation is a powerful emotion and addictive. 

It leads to a lot of victim-blaming. If those people didn't make us mad, then they wouldn't have to deal with the consequences of making us mad. This is straight out of the abusive spouse/parent playbook. It feels righteous, it sure seems like righteous indignation -- while you're in it. Then come the apologies and then another relapse.

It justifies acting badly. When we use force, bullying, pillory, public humiliation, verbal abuse, defamation ... well, that's okay because we are the good guys. It's okay when Batman takes the law into his own hands. It's okay when Superman drops someone's office building on Luthor. It's okay when we bomb cities where bad guys are hiding. It's all fine because good guys get to do bad things in the cause of justice.

Although, it's hard to tell the good guys from the bad guys when their methods are identical, isn't it? 

An unexamined righteousness is a dangerous thing.

What if our anger is only presenting as righteousness and comeuppance, but is based on a misread of our circumstances or other people's motivation? Where is the curiosity, openness, discussion, dialogue?

How fast are we to label a villain and search them out for signs of villainy?

And what if we're only a little right, and the so-called villain is not set in their way but willing to be converted? Does our crusade drive them into greater "villainy" as it feeds their sense of being victimized unfairly? Doesn't it activate their righteous indignation?

Oh, that sounds like today's social media. And all other media too. 

Welcome to the age of outrage.

"Social" media (so-called, though it tends to be a shouting factory)  is full of people sloppy drunk on outrage.  Often, being angry and venting that anger is all the action that is taken. Abusing and bullying to "make them suffer for their crimes" masquerades as helping.  It's likely more about revenge and sharing pain than making a change, and unlikely to create any change. It's pretty good at creating animosity, broken relationships, lost opportunities, and enemies. So, you know, if that's "a better world" then I guess it is positive action.

I have found that outrage, especially habitual and recreational anger, make my life less creative, happy, useful, and meaningful while also not leading me to make any real change in the world (only enemies).

This is why I have been on a multi-year journey to rid myself of habitual anger, recreational anger, and worry.  

Strangely, I find I'm more likely to take action out of compassion for others than out of outrage and anger.

When I'm angry, I crave a victim and an opportunity for a violent act (even if it's just angry words).

When I care, I look to help.

I know it's not my job to tell other people how to think and feel. So I'm just sharing what works for me and inviting everyone who is of a mind to listen and consider alternatives.

Maybe we could all be less sure, less angry, and more busy looking for allies and partners and friends rather than scouring social media for enemies to shout down.

Sure, that's sensible, though not everyone who is sloppy drunk on outrage is looking to be sensible.

I don't know how to talk sense to a drunk. I have to instead look for opportunities to talk to the sober, or those who are looking to become so. Looking for kindred souls seems better to me than shouting down angry people.

You've come this far, so maybe you and I have just found each other.

How are you doing? How can we get to know each other? What interests you?

How did you escape your outrage and start to take responsibility for yourself and others?
Do we need a hashtag?

Tuesday, June 26, 2018

Organizational Dysfunction

For decades now, people are taught a definition of "success":
a successful team delivers the full scope on time and under budget.

It has been the mantra of project managers and has been taught in books, in magazines, in college classes, and by word-of-mouth for GENERATIONS of managers and workers.

Starting in the 80s and 90s we started hearing echoes from the 60s that a successful project is built as a series of small successes with incremental and iterative work.

Next, we hear that a project is successful if it meets the client's needs on the day of delivery, rather than the day the contract is signed.

And then we hear that successful products are different from projects, and project-thinking may not even be valid. The goal is to build a user base, not a defined content. Now it is about the income (economic engine) that the product represents.  No more thinking about the day of delivery and the payment for delivering the thing; it's a longer view.

And then we have Lean Startup telling us that we may have to pivot in order to develop customers; change the product, or find it a new audience. To not be married to content.

This awareness and understanding are not new (seriously, much goes back to 60s and 70s) but is "not evenly distributed."

Some people are pursuing eighties.success and seventies.success still, and haven't gotten the memo.

We tend to refer to un-updated-ness as "organizational dysfunction" It used to be "organizational function" -- even "successful project management." There is also some amount of process and HR cruft that builds up in the organization to hold the older view in place and keep the people successful (to the old definition). This we also call "organizational dysfunction."

In the interest of "try not to be a threat to the people you're trying to help" I suggest that we change terms.

The people are not (usually) evil or broken or dysfunctional people.

The org has a great potential.

There are just new ways that may help a company be more successful. There is a new view of working, and what it means to be successful. We think these are better views. They seem to be.

I don't know how it helps to insult the people we serve or to war against "wrong-headedness" instead of instilling a new way of seeing work, success, our roles in this new way of working.

This is a long-winded way of asking that we who consult and train and transition maybe try "cranking up the good" instead of "decrying the dysfunction" for a while and see how it turns out for us.

"Organizational Dysfunction" maybe is just "not-yet-transitioned-ness" and maybe that's why we are here. 

Monday, June 25, 2018

Agile In A Nutshell

I had answered a question on LinkedIn, after which a friend was kind enough to quote my answer on twitter with a kind recommendation.

I decided to reproduce that answer here. Enjoy!


The Post


"We have initial idea for a change but approach and end result is not clear.
The approach and overall requirement tend to change during the course of development." 

Congratulations! You are in exactly the standard circumstances for a truly agile approach! The secrets here are to be incremental (adding bits at a time, rather than coding to a designed end-point) and iterative (revisiting code to improve its design and function) using inspect-and-adapt.

The standard rule is "make it do something, however trivial, on day one; then keep it running at all times" and "add functionality in very thin end-to-end bits so that you don't have a big integration nightmare at the end."

Enemies of this approach are anything that makes it hard to iterate, increment, or integrate. If the document gets in the way, then drop it. If it helps, then use it. But remember that the primary measure of progress is working, tested code.

Thursday, June 14, 2018

Owner, Renter, Guest, Vandal?

What is your role in the code base you find yourself in today? 
Owner? 
Renter?
Guest?
Visitor? 
Vandal?
I ask seriously because I'm learning that the answers matter. 
A lot.
An owner makes property improvements, upgrades, installations, replacements.  It belongs to them, and its value is important. An owner can paint, can put in a staircase, can replace the plumbing.
A renter tries to not make things worse and can make superficial improvements. Maybe they can paint or change out some things, but only with permission from the owner. Anything that is really wrong, that's on the owner.
A renter doesn't repave the drive, put in a deck, fix the drainage of the basement. It's not really theirs.
A guest is just there for the day; not really permitted to make changes at all. They can use the facilities that exist, but that's all. Could you imagine, "Hello people of Pisa. I just noticed your tower is crooked, so I fixed it."  Guests are just passing through.
A vandal doesn't care about the quality and state of the property at all. They are just there to put their personal mark on it and they hope to avoid the consequences of doing so. 

You?

How do you treat the code you're in? Can you make the changes you want to make, so that it becomes more valuable? Can you make any substantive changes without permission? Do you just get the effect you want and get out before you're caught?
Is it a place you live, or a thing you wander into and out of sometimes?
There is another possibility: contractor.
If you're playing the part of a contractor then you're there to make a specific change on the behalf of the owner. 
The property isn't yours, and you're not invested in its value. You may take pride in your work and reputation and therefore refuse to do slapdash work. A great contractor acts in the owners' best interest.
On the other hand, you might be the budget option, the lowest bidder. There is a pretty small difference between a budget (slapdash) contractor and a vandal. 
Which are you? 
Which do you want to be? 
Which are you expected to be?
Does a misalignment in the expectation you have and the expectation others have of your role create problems?

Wednesday, June 13, 2018

Piecing it together without tests...

I wrote a thing yesterday…. it was in python, and I was using PyCharm (which is very nice, and a departure from using vim + sniffer).

It was something I didn’t know how to do initially, and couldn’t really figure out how to test drive.  I did it without pair- or mob-programming.

It was essentially a screen-scraping tool to dl the latest version of a zip file... that's not really interesting.

I had to learn or relearn some libraries I’d forgotten about, using docs and StackOverflow, and the “inspect element” feature of chrome quite heavily.  I whinged about the lack of ids and names on elements in web pages on Twitter, but eventually got it worked out.

Because I wasn't sure how to make it work to begin with, I used the ipython repl (via the 'Python Terminal' window in PyCharm. I would try various ways of locating the anchor link I wanted until I found one that I thought was far less-fragile than the other ways (such as some rather dubious XPath). I basically pasted the lines that worked into the source file.

I ended up using requests, BeautifulSoup, and urlparse to do the whole job. It was really simple once I figured it out.

When I was done, I had something pretty ugly and linear.  I refactored it into composed functions and re-ran it to test it.

At that point I realized that I’d not really written any tests for it — I was building it in the repl mostly, and then pasting lines to the source file.  It dawned on me that a lot of code is written this way -- tested using the actual servers (not mocks or local data), pieced together from documents and code snippets and trail-n-error, tested only in the sense of being test-run by hand, and done entirely alone.

Since I didn't have the disciplinary reinforcement of a programming partner, it was easy to fall into making something that wasn't driven through automated tests, just through "try and see."

As my Industrial Logic coaching partner Mike Rieser would say, I was "programming for effect only."

I want to be fair about this being a little throwaway utility. It will be easy to discard and rewrite or to fix if the page changes in an unfriendly way. It has composed functions. It's tiny -- less than 50 lines including vertical whitespace. Easily 90% of the work was just learning how.

But I didn't test drive it. I didn't use partners. I could easily have done both.

I'm realizing that Lewin's equation was right. My behavior is a function of both my personality and my environment, and by taking myself out of my normal environment my behavior was not at the same professional level as usual.

Something to think about.