Are You Doing a “NASA?” The Perils of Mental Models
Decision-making is the means by which we most directly shape our lives. Some decisions we make consciously; many more are “automatic” decisions made in response to stimulus. The first type of decision-making is a distinctly human domain. Other species don’t have the same breadth of cognitive tools, and therefore can’t be described as true decision-makers.
If we want to improve our life results and get more of what we want, we must make different and higher-quality decisions. The other choice is to keep doing the same things over and over or hope that someone or something will come along and help us out. This is called Magical Thinking. It isn’t helpful.
Most of us are not as good at decision-making as we think we are. Why not? One of the biggest reasons is our succeptibility to influence by the people around us, something writer Laurence Gonzalez calls “groupness.”
The idea is simple: When we are in a group setting we tend to turn our brains off and take our clues from the people around us. Think you don’t? You do. Consider Gonzales’s story about NASA and not one but two space shuttle disasters.
After the space shuttle Columbia broke up in flight on February 1, 2003, a commission was formed to investigate the accident. It covered all the mechanical and physical facts of the explosion, but that left a very basic and vexing problem. The accident had been avoidable and had not been avoided. That meant, in effect, that the smartest guys in the world had done the dumbest thing in the world. Twice. The commission’s report sought to explain how this could happen by talking about the very sorts of mental models and scripts, along with group hostilities, that shape the lives of the Rattlers and Eagles of this world and that shaped the experiences of John Tanner and Homo erectus as well.
Managers at NASA had fashioned a psychological framework that allowed them to systematically ignore clear evidence that they were heading into trouble. NASA’s triumphant experiences in putting men on the moon during the Apollo program of the 1960s had led to the formation of mental models and behavioral scripts in the organizational culture that persisted despite drastic changes in the environment, such as greatly reduced budgets and overwhelming evidence that essential pieces of equipment were malfunctioning.
The second force influencing critical decisions at NASA was groupness. The final report of the commission on the Columbia accident said, “External criticism and doubt…reinforced the will to ‘impose the party line vision on the environment, not to reconsider it….’ This in turn led to ‘flawed decision making, self deception, introversion and diminished curiosity about the world outside the perfect place.’”
The report is quoting Garry D. Brewer, a professor of organizational behavior at Yale University, who was attempting to explain how management at NASA could have behaved the way it did. The “external criticism and doubt” came, for example, after the explosion of the space shuttle Challenger in 1986, the first time that NASA made the worst mistake it could have made. That criticism came, significantly, from outside of the in-group.13
The combination of groupness and persistent mental models made for an organization that could not take in new information when that information did not accord with its indelible concept of itself as the “perfect place,” as Brewer called it. Moreover, it could take any contradictory information and reinterpret it as confirming the existing model. This came about through two major influences that made NASA’s models unassailable and made its culture hostile to all outside groups. To begin with, there was the unprecedented investment not just of money but also of personal and emotional effort during the Apollo program. The divorce rate was high, as marriages fell apart under the strain. People literally gave their lives for the effort. Gus Grissom, Roger Chaffee, and Ed White died in a fire during a test on the launch pad in 1967. But following all that sacrifice was an astounding success, arguably the highest achievement of human technical culture, with men walking on the moon while we watched them on television. This has all the ingredients necessary to form robust models and scripts (big investment, big reward). And that experience simply hardened the shell of groupness that already characterized NASA.
NASA’s unspoken and unconscious attitude by that time was: We must be right; after all, we put a man on the moon. There had been many reinforcing steps along the way, too. During Apollo 13, for example, the concept that “failure is not an option” was developed, and the safe return of Apollo 13 served to strengthen the models and the ability of groupness to repel ideas from outside. It also promoted a dangerously wrong idea. For failure, unfortunately, is always an option.
The big problem here is that successful outcomes, broadly defined, caused people to ignore disconfirming information, out of spec performance, and dysfunctional decision processes. Or to put it differently, if nothing bad happens, we must be doing something right. More . . .
As the investigating board put it, “Both Columbia and Challenger were lost also because of the failure of NASA’s organizational system…. Both accidents were ‘failures of foresight’ in which history played a prominent role.”
In the case of Challenger, engineers were faced with the fact that fuel in the solid rocket boosters was burning through the rubber O-rings that sealed the seams where sections of the rockets were joined. Groupness dictated that no one outside that immediate culture was fit to judge the fruits of their labors. Confirmation bias is a phenomenon in psychology by which people tend to take any information as confirmation of what they already believe. In addition, they tend to ignore or miss any information that doesn’t confirm what they already believe. This can work to gradually revise a mental model in a one-way direction. Because NASA believed that “we’re the best” and that “failure is not an option,” all information tended to support that conclusion, no matter how contrary it might have seemed to an outsider.
Each time the solid rocket fuel burned the rubber O-rings during launch without an accident happening, the engineers at NASA readjusted their models and scripts slightly to accommodate that as “normal.” Through a subtle progression, a complete failure of design was turned into an acceptable situation. Each time nothing bad happened, they did it again. This confirmed the mental model, even while groupness helped to keep conflicting information from having any effect. The weather was the spinning roulette wheel in this complex system that NASA managers had unwittingly set up for themselves. All they needed was for cold enough weather to coincide with a launch, because cold made the rubber O-rings more brittle and therefore more likely to burn through. It was just a matter of time. The ape-like hierarchy at NASA ensured that those engineers who knew or suspected the truth would not be heard.
What makes this whole story especially sad is that NASA learned nothing about decision-making, the root of the problem.
The same array of troubles bedeviled Columbia. Insulating foam blew off the main fuel tank and hit the orbiter. The engineers had seen it happen a number of times, but management kept on launching anyway. When nothing bad happened, they took that as confirmation that they were right and reset their mental models to accommodate the malfunction. As the final report on the accident clearly stated, “The initial Shuttle design predicted neither foam debris problems nor poor sealing action of the Solid Rocket Booster joints. To experience either on a mission was a violation of design specifications. The anomalies were signals of potential danger, not something to be tolerated.”
But in the culture that had evolved at NASA, each return from a successful mission was another moon landing. If the world had largely come to ignore space launches, NASA was still hearing applause that was, by the time of Columbia, more than thirty years old. So, instead of peering more deeply into the problem, they gradually revised their models until they were literally interpreting failure as success. The final report of the commission said:
Engineers and managers incorporated worsening anomalies into the engineering experience base, which functioned as an elastic waistband, expanding to hold larger deviations from the original design. Anomalies that did not lead to catastrophic failure were treated as a source of valid engineering data that justified further flights.
The moral of the story . . .
And this is precisely how a mental model can be expected to function. It operates on a simple rule: if nothing bad happens, you must be doing something right [emphasis added]. So influential were NASA’s models and scripts, and so delusional its self-confidence bred of groupness, that even after Columbia broke up, killing all on board, the space shuttle program manager told the press that he was “comfortable” with his previous assessments of risk and didn’t think the foam debris had caused the accident. But remember that a key feature of this system is that, taken one small step at a time, each decision always seems correct.
Tags: Challenger, Columbia, Decision Making, Decision Quality, NASA, O-Rings