In 2010 during the BP Oil Spill catastrophy in the Gulf of Mexico David Brooks of the New York Times on the PBS News Hour made an observation about complexity:

DAVID BROOKS: One of the things that has happened is that we now base our lives and our civilization on incredibly complicated systems. The financial market is incredibly complicated, drilling holes in -- miles down in the ocean, incredibly complicated systems. And we charge people to run these systems, when they really can't evaluate the risks very well.

They are so complicated, they are beyond comprehension. So, if you look at the decisions that were actually made on the platform in the hours up, they had some tests which they didn't know whether they were right or not. Should they recirculate the mud in the right way? Should they cement the hole? Did they do that in the right way?

And you had people making decisions based on certain risk assessments which were completely wrong. And we have seen this before. We saw it in the financial markets. We saw it in the Challenger explosions. People were -- we're based on incredibly complicated technical systems that we do not understand.

And in these circumstances, we are often vastly overconfident about how much...

JIM LEHRER: How we can do that.

DAVID BROOKS: ... we can control. And people take risks, and you get disasters.

Brooks makes a compelling, if somewhat skeptical, case that we should expect failures because of complexity. I don't share Brooks' skeptecism because I know the nature of engineers, testers, and IT people on the technical team. We are problem solvers. We love to get to the root cause of a problem and mitigate.

Experience shows me that complexity impacts an organizations management more than the technical team. I get to work with hundreds of CIOs, CTOs, software engineering managers, and operations managers. Management needs an ambit:

1. Confidence in the past performance of the technical team
2. A dashboard showing metrics of a system's performance, including performance trends (does it run faster, quicker, less expensively now?) and a count of failures
   
3. Testing coverage: A measure of how many parts of the system are being testing, and how often
   
4. Compliance against specifications: How often is the content of the inputs and outputs consistent with the specification?
   

Unfortunately most of the technical team members are unaware, forget, or just-don't-want-to provide management with help determining the ambit. Technical team members advance their careers when they treat each work each day as time to provide management the ambit.

I agree with Brooks on the topic of responsibility. We need to rise to the challenge to bring control and managability to the systems we use. It would be one thing if complicated systems failed by design, and in a way that is both controllable and manageable. Most of the time I am dismayed to see these systems fail with no plan, no analysis, and no way to avoid the failure in the future.

Instead of thinking that an organization the size of British Petroleum needs to worry about complexity, we can take an audit of our own behavior on an individual scale. How much complexity are you willing to risk and bear? I had an experience recently at the airport security line that illustrates the problem.

My work requires me to travel frequently. I often carry two laptops and a variety of equipment. I put my backpack onto the airport security conveyor belt and watched the agent try to make sense of the contents. The following photo is my backpack after I removed the two laptops.



My backpack is a complex system. I often have to take it apart and explain how it works. I need my complex system to understand the functional issues, performance bottlenecks, and tools capabilities to test information systems. It is a complex system, a system I accept being charged with, and I can evaluate the risks very well.

-Frank