Musk himself acknowledged that figuring out what happened was the most difficult challenge SpaceX had faced since starting the (at the time) truly crazy project of building a completely new rocket from scratch. The issue was compounded by the fact that they’d had an in-flight failure in 2015 (flight CRS-7), also involving a RUD of the second stage. That one was particularly bad, since it was also an ISS resupply mission and the Draco spacecraft was carrying in its unpressurized trunk one of the docking mechanisms (IDA — International Docking Adapter) needed for the new generation of manned vehicles being built. Fortunately, there was a second IDA almost ready for flight and that one was completed and successfully delivered to the ISS by a Falcon-boosted Dragon on CRS-9 in July 2016.
The CRS-7 investigation determined that the accident was caused by failure of a strut holding the liquid oxygen (LOX) tank in the second stage. The strut was manufactured by an outside supplier and inspection of others in the supply chain showed problems with it. There were those who thought the investigation into the CRS-7 accident was too hasty and not sufficiently “managed” by NASA. [Insert tedious discussion of aerospace accident investigation, flight safety criteria and spaceflight risk management.] In the aftermath of the September 2016 “firexplanomaly,” the army of SpaceX critics piled on, claiming that the CRS-7 investigation had been rushed and flawed, and that the second failure of the second stage could have had the same cause — and therefore that the flawed strut was not the problem.
To make a very long story short, SpaceX concluded the problem in the September 2016 pad kaboom was a failure of a pressurized, super-cooled helium tank. The helium is used to maintain pressure in the LOX tank as it is emptied during flight — a mechanism common to most modern rockets. But there are two things about the Falcon second stage helium tanks that are unusual. The first is that it is placed inside the LOX tank. The classic approach for pressurizer-tanks is that they are located outside LOX or fuel tanks. The Falcon second stage helium tanks are located inside the LOX tank to save space (and therefore mass), and also to use the (super-) low temperature of the LOX to allow the He tank to hold more because it is chilled by the surrounding LOX to such an extent that the He starts out as a liquid — that’s REALLY cold.
The second and crucial factor about the He tank is that it is made with COP — Carbon Over-Wrap. Again to save weight, the tank has an aluminum liner surrounded by a carbon fiber and epoxy overwrap for strength. There is a long and (for the truly geeky) fascinating history of people trying to use carbon fiber composites to make fuel, oxidizer and pressurant tanks for rockets because of mass savings. That history involves a lot of heartbreak. Basically, we understand metallurgy a lot better than we understand “carbonology.” The people at NASA and the aerospace contractors who have been pushing this technology for a couple of decades have been on the bleeding edge and have encountered some spectacular and costly failures in the quest to use composites to lower tank weight. Major rocket projects have been cancelled because of failures along the way on the quest for carbon composite fuel and oxidizer tanks.
Again to make a long story short, SpaceX concluded that the September 2016 accident was caused by failure of the helium tank carbon overwrap. They believe that during propellant loading, LOX worked its way into the carbon weave and solidified into oxygen ice, causing slight carbon fiber delamination, which caused the He tank to bulge and fail. Basically, the helium tank popped like a balloon, causing the LOX tank to rupture, causing structural failure of the second stage, causing KABOOM!
The real systems head-scratcher came when trying to explain why the failure occurred on that flight. The COP helium tanks had been used successfully on a number of previous flights. The explosion was so drastic that there was no debris to inspect. Other tanks in the line had no visible flaws and there were no manufacturing records showing issues with the failed tank. Shortening the story once again, the conclusion was that the propellant-loading procedures on the September 2016 flight were slightly more aggressive than on previous flights. This raises a fundamental issue which is a subject of constant discussion in the space policy-geek community about Space-X’s approach to system development: One of the key elements of how SpaceX has managed to accomplish so much in such a short time is by constantly upgrading and tweaking its designs and systems. This is in sharp contrast to how NASA and the traditional Big Space / Old Space contractors have come to approach spaceflight over the period since Apollo and the beginning of the Shuttle era. NASA and its bloated cost-plus contractors have become EXTREMELY conservative in how they approach systems and procedures of every kind.
Again, there are many books’ worth of discussion about this, and the conflict over Space-X’s more aggressive approach to flying-while-innovating has sometimes been extremely intense. But despite critics alleging that Musk takes a cavalier attitude toward risk management, what SpaceX did on this occasion was squarely within the traditional rule-book: They went back to the propellant loading procedure they’d used previously (and also began a longer term redesign of the helium tanks — but that will take at least a year and maybe longer to get to flyable hardware).
The successful launch on Sunday was actually the second Falcon 9 flight since the “firexplanomaly.” On January 14, a Falcon 9 launched from Vandenberg AFB in California, lofting a set of ten next-generation Iridium communications satellites into near-polar orbit. (And the first stage nailed a drone-ship landing in the Pacific ocean, as well.) SpaceX critics have constantly harped on the need for the company to achieve a track record of a long series of successful launches with a stable design. Given the ambitious goals and develop-while-you-fly philosophy of the company, SpaceX supporters (and I am definitely — and very obviously — one of those) celebrated the CRS-10 flight on Sunday as what counts as a “track record.”
So, why was the fact that the launch took place from pad 39A important? There are three reasons. One is the obvious historical factor that 39A was the departure point for Apollo 11 and many shuttle flights (including the first one). Second, though, is the fact that the September 2016 RUD did a number on LC-40 (“LC” = “Launch Complex”), the pad SpaceX has been using for all previous KSC launches. LC-40 won’t be back in business until late this year at the earliest. To stay in the game, SpaceX absolutely has to be able to launch from KSC to reach equatorial or other lower-inclination orbits. Whether the company could have weathered a delay in low-inclination flights until late 2017 (i.e. survived commercially) is a Big Question. But Musk knew for a long time that he’d need at least two pads at KSC, not least of which because, as they say, “space is hard,” and rockets do blow up. So the 39A lease had been in the works for years, and the effort to transform it into a pad able to support the Falcon family of rockets was nearing completion at the end of 2016.
Note that SpaceX is developing a completely new launch facility in Brownsville, Texas which will be able to make launches into low-inclination orbits, but that won’t be ready for some years. When the Brownsville facility comes online, things will really start to get interesting, since first stages launched from there will be able to land in Florida at KSC. This will require less fuel for the landing than the current boost-back does — thus increasing the useful payload to orbit, and will create a “virtuous cycle” of the two facilities supporting each other in multiple ways. At that point we should start to see what the whole SpaceX endeavor is really about: radically lowering the pound-to-orbit cost.
The third and most important reason this launch location is significant is that the two LC-39 pads are the “crown jewels” of the KSC launch facilities. They were originally built to accommodate the Apollo-era Saturn V and were then modified into Shuttle launch pads. They are massive works of civil engineering, supported by an (aging but still serviceable) infrastructure of chilling, storage tanks, pumps and plumbing necessary to support cryogenic rocket propellants. “Letting go” of these and turning them over to commercial operators was a big step in NASA’s recovery from its bureaucratic malaise. SpaceX now has exclusive use of 39A under its long-term lease. Pad 39B is reserved for primary use by NASA’s own rocket/manned spacecraft program, the so-called Space Launch System (known by its detractors — including me — as the “Senate Launch System” — a program having a primary design criterion of employing as many contractors in as many congressional districts as possible, and one that has offered yet more evidence that NASA should not be in the business of developing rockets and spacecraft other than for purely scientific or technology-development purposes). But the re-work to make 39B suitable for launching the SLS is being done in a way that makes that pad maximally flexible, and NASA has already granted rights to use it to Jeff Bezos’ up-and-coming rocket and spacecraft company, Blue Origin. (Note that there was a major stink fight between Musk and Bezos over the rights to use LC-39 facilities but, in the end, both got what they wanted — isn’t competition great?)
The icing on the cake for Sunday’s launch was the return to launch site (“RTLS” for space nerds) of the F9 first stage. This is getting to be routine now. Before long, SpaceX will refly one of its recovered first stages, and with that, we will be able to say that we have well and truly entered a new age of space development.