Thursday, September 17, 2015

Confirmation Bias and The Twelve-Day Technical Problem

Charlie is a developer. He's just been assigned/volunteered to take on an interesting technical challenge. It will involve research, learning, and experimentation so it is exactly the kind of assignment that most developers would love to have.

If the solution to the problem was clear cut and obvious, then it would already exist in a library or a book or in google or stackoverflow. It doesn't, to the best of Charlie's knowledge. Of course, he looked. This isn't Charlie's first rodeo.

Vinnie is Charlie's manager. He's a smart fellow, and has quite a bit of experience. He has a good sense of organizational dynamics and deadlines and an amazing memory for minutiae that escapes other managers. He can juggle tasks and schedules like a wedding planner, and never misses crucial dependencies. He knows all his employees' spouses and children by name.

Vinnie's sense of deadlines and productivity naturally enough led him to ask Charlie how long this technical task will take. That's a reasonable question. The only problem is that Charlie doesn't really know how much research or experimentation might be necessary. He guesses that 10 days will be enough, and Vinnie marks the date on his calendar.

Monday is retrospective and sprint demo, so Vinnie and Charlie are very much involved in that work. Tuesday is planning meetings and the beginning of the sprint. Charlie nearly signed up for a sprint backlog item, but Vinnie (ever watchful) reminded him that he had a special task already assigned.

Fast-forward 6 days. Vinnie comes to see if the task is half done. Charlie doesn't know. He's tried several things that didn't work, and has dropped two promising-seeming tools that can't really handle this particular challenge. He's got a new design.  Vinnie asks if it's hopeless, and Charlie lets him know that he's vectoring in on an answer and once he knows what to do the implementation will be quick. Vinnie checks his calendar and feels a rising nervousness.

Day 7 comes, and there is no solution. Charlie's abandoned a couple of more designs and has got an example in proof-of-concept form that handles most of the cases they'll encounter in production (according to the data set they've generated) but won't cover several very crucial examples. Charlie says some more digging is necessary.

On day 9 Charlie still doesn't have that last little bit done. Vinnie reminds him of the deadline agreement. The excuses begin to flow. The problem is "hard." The solution is "novel." It is a "research problem" and he shouldn't be held to the deadline because 10 days was just a "guess". Besides, a day and a half of that time was dedicated to sprint activities and Charlie was still responsible to teammates for technical advice on other parts of the system.

Vinnie turns red and leaves the room to ponder. On his way to the coffee machine in the afternoon he sees Charlie playing foosball.  Vinnie realizes that Charlie is just not taking this seriously enough. As a manager, Vinnie understand how to motivate people.

Vinnie sends Charlie back to his desk and vows to keep his eye on the recreational equipment for the next day or two. Charlie was hoping that if he could just take his mind off the problem for a few minutes, he could return with mental clarity. Instead, he just trudges on in a fog.

Day 10 comes.  Charlie is talking on the phone to somebody. Vinnie demands that Charlie put the phone down and get back to work. Seeing Vinnie's mood, Charlie offers no resistance. He shrugs, hangs up, and turns back to his computer. Vinnie walks away satisfied that he's made it clear to Charlie that he's serious about this work. He's "set him straight."

Charlie had been conversing about the work with a colleague who had mental freshness and a different background. Had the conversation continued another 20 minutes, it could have saved Charlie at least a day's work.

Day 11 comes and goes. On day 12 Vinnie is seen yelling at Charlie, not mincing words, using strong language, letting him know how much is on the line and how important the work is, and how Charlie needs to focus and take it seriously.

Charlie is frustrated. He's been working hard the whole time and suffering "abuse" for the last several days. Now he's very nearly done, but he's standing in Vinnie's office listening to a motivational talk about the finishing the work when he could be finishing the work.

It's frustrating, but of course Vinnie is in the position of power and in no mood for counter-argument. Charlie holds his tongue, and agrees to focus and keep working for an answer.

He agrees to send bi-hourly status reports via email. Each takes about 15 minutes to write and edit to make sure that he doesn't accidentally trigger Vinnie's anxiety. Each distracts him from the work he's trying to do and focuses him on the frustration and rising anger he is feeling.

He works late that night to make up for the time he's spent trying to placate Vinnie.

Day 12 and Charlie is fixing the mistakes he made because of working into the night last night. Again a visit and lecture from Vinnie.

This time, Charlie allowed his mind to wander (as tired as he is, focus is difficult) he realized exactly what was needed to get that last little exception handled. He waits 20 minutes for the lecture to end and codes up the rest of the solution. By 4:20 in the evening, it works for all the cases in their data set and he pushes the code to QA.

Duelling Conclusions

Lessons learned here will be generalized and incorporated into the fabric of "what work is like."

To Charlie this is a story of having to endure ridiculous motivational tirades in order to fix a problem that, fairly, would have taken 10 +/- 2 days not matter what. It might have taken fewer days if it wasn't for the manager's interference, but it was still finished more-or-less on time.

Charlie's lesson learned: interacting with managers is distracting, unpleasant, and fraught with political peril. He will avoid Vinnie as much as possible in the future, and placate when avoidance isn't possible.

This is, of course, an awful theory of how people on the same team should work together, but from someone with little organizational power, it is the best he feels he can expect.

To Vinnie it's clear that the work wasn't being done until he started really pushing and leaning on Charlie, but once he sufficiently motivated him it was done in a couple of days.

Vinnie doesn't see it as a 12-day technical issue, but a ten-day motivation problem followed by two days of "real work." With the realization that developers are lazy and self-indulgent, he now knows to keep pressure on his developers so that they'll do their work instead of goofing off.

Vinnie is suffering from the worst kind of confirmation bias, and will come to believe that abusing his people is the best way to get work done on time.

The stories are misaligned, and the take-away lessons are in strong conflict. There will be a downward spiral if neither viewpoint changes.

In this way, the world keeps turning and the stories we tell perpetuate bad theories about how people should work together. The gap between us widens with the social friction and mutual condemnation. How long before we lose all respect for each other?