Friday, April 26, 2013

Toward a New Theory of Productivity

Standard theory says that productivity is output over input. That's easy enough to grasp, and entirely right.
 p = o / i

Of course, input means a whole lot of different things. A cabinet-maker who puts in 20 hours of labor every day but has no stock of wood or tools would be unproductive. Likewise, a pile of boards and hardware is unproductive without a skilled laborer to fit them together.

Output is also multidimensional. It involves quality as well as quantity. An inept carpenter who thoughtlessly half-drove nails into two boards, called the result a cabinet, and shipped it would be productive in quantity terms, shipping dozens and dozens of "cabinets" a day,  but his output would be unusable waste.

This is all dead obvious until you bring up software.

Software is weird.

Software doesn't involve any visible raw supplies, has no stockpile of parts, requires a rather small bit of easy physical labor (typing), and produces a product that is mostly invisible.

What are the real inputs and outputs?

There is an unenlightened view that software productivity is a simple function of effort. 
p = f(e)

It's not entirely false (if you don't bother trying, you get nothing done) but it's horribly and embarrassingly naive.

I've observed that the function collapses under sufficient values of e. Too much effort, too little rest, too much push-push-push, and far less gets done. For high values of effort, the work that gets done tends to be regrettable (see the unskilled carpenter example, above).

It's not the quantity of work that matters so much as the quality of work. This is "obvious" some say, but observe how much the quality of work is ignored in the effort-maximizing theories of less enlightened software managers. We can do better.

A slightly more enlightened view is that it's not pure effort, but focused effort. If we have twice the focused effort, we should be getting some percentage more work done, and have higher quality.
p = f(E)

Slightly more enlightened (or reality-based) is that productivity involves effort over time. Too little effort leaves money on the table, and too much effort reduces effectiveness.
p = f(E,T)

By the way, I would love to find just one or two managers in a large corporation who have tracked productivity against effort and have found the sweet spot where either increasing or reducing effort will cause a reduction in production.

In corporations, there is seldom any real effort to quantify and optimize. Instead, demanding overtime and additional staffing are seen as a "show of force." The underlying assumption about force is that if you aren't getting what you want, you aren't using enough of it. It's unenlightened, but it sure looks good to be seen flexing your muscles.

This set of theories is embarrassing because each ignores non-labor inputs.

Let's try something different.

Many years ago, Dr. Ralph Johnson (professor at UIUC, co-author of Design Patterns among other things) told me that frameworks are just distilled experience. I thought it sounded profound at the time, and agreed with him that where the framework is ugly is where ugliness is experienced; workarounds have causes and are hard-won.

It took me a long time to realize that all software is distilled experience.

Then, only a short while ago, I was talking to a programmer named Ray Scheufler who told me why it's not effective for someone to hand a developer a design to implement. As best I can recall his words, they were
"When I'm programming, I have to make a lot of decisions. If I don't know why the design is formed like it is, then I can't make those decisions competently."

This starts to point us into a new direction.

If software is (as I posit) distilled experience, and programming is (as I further posit) the act of making hundreds or thousands of decisions, then effort is not a primary factor.

Having focused effort is still important, but increasing effort when you are lacking other inputs will be ineffective.

If we accept that this labor is intellectual, and primarily consists of thinking and making decisions, then we have decades and centuries of literature to pull from to design a much better theory of productivity.

There is much already written about attentiveness, focus, decision fatigue, social threats, ideal thinking conditions, supporting practices, etc.  I think that David Rock has presented a lot of the important bits in his books 'Your Brain At Work' and 'Quiet Leadership.'

Likewise, conditions for learning and growing are well-explored in Dr. Carol Dweck's book 'Mindset.'

We are not in unexplored territory.

Programming is thinking, so a theory of productivity would be based on making it easier and faster to:

The ability to not over-invest in unprofitable work requires a great deal of process, information, and wisdom -- these don't come from mere nose-to-the-grindstone effort.

I hold that primary inputs for software development include: 
  • interest and mental freshness -  motivation, lack of decision fatigue, etc.
  • availability of relevant information  - technical details of the platform, language, libraries, designs as well as business details and expectations
  • confidence - consensus, experience, supporting documents, community
  • feedback - tests, reviews, demos, etc.
This is not a fully-formed theory of productivity, but I believe it will lead us closer to one. I welcome all who are interested in contributing.