Hydro Testing Lessons

Earlier this year, I began my rocket engine testing campaign, and began by hydro testing my tanks. During my testing I ran into numerous problems, one of which was a major hardware failure and a few other smaller mistakes that led to the hardware failure. In the aftermath of this failure I learned so much about weld engineering and testing procedures that it made the failure seem to be worth it. Granted it set me back but I don’t need that tank for a while now. 

Hydrostatic testing is filling up a pressure vessel with water and then pumping more water into a closed system to raise the pressure, converting kinetic energy (pumping action) into pressure head. This is done with water because water is a (mostly) incompressible fluid and therefore upon failure, there is not a sudden rush or explosion, typically it is a small leak and loss of pressure. This makes it possibly the safest fluid with which to do static testing, air would be straight explosive and everything else is just mostly inconvenient. My test apparatus changed over the course of my testing, generally towards the better and always involved basically the same plumbing setup. This included a hand pump connected to a one way valve connected to the tank and a drain solenoid for draining the pressure and a pressure gauge or transducer reading the pressure. Complete drain of the tank would have to be done by hand - by physically tipping the tank over and letting it sit for a bit similar to how filling the tank would be done via a funnel which is much much faster than manually pumping in small volumes of water. A pump like this converts a given mass flow of water from a high volume low pressure environment into a low volume high pressure environment. Originally the pump I had been using had been used to pump hydraulic fluid on airplanes somewhere somehow. That’s all I could surmise from the ebay listings of the guy I bought it from. Still an interesting history. Getting to the first test, I had first forgot to get any o-rings to seal some of my o-ring boss fittings and so I had numerous leaks all over the place which wasted the first day of testing, then I got the o-rings, got some teflon tape and sealed up everything, and then on the second day I was finally able to build pressure after replacing a faulty drain valve which could not handle the given pressure. I filled the tank with water and then pumped more water into it while it was open to bleed the air from the system. Then having done that I sealed off the system and began building pressure. Everything seemed to go nominally except for the pressure gauge which was showing inaccurate readings. However we simply measured the pressure rise which we figured to be accurate. It was clear that after this first test that these pressure gauges would not be useful for this application. We kept building pressure till we thought that the tank had passed our desired pressure and passed the test. With that the helium tank was a probable success. Later findings threw some doubt in the face of those results due to the inaccurate pressure readings, retesting will occur next month. 

Test two was a bit more chaotic. Basically the same procedure as the first test except I had just wired a brand new DAQ system and hooked up a new pressure transducer to it instead of the previous pressure gauge. This was my first time playing with pressure transducers and I didn’t realize that maybe it was calibrated wrong or my pressure readings were off. We find out after pumping for a while that the tank failed - dramatically and with a boom, splitting most of the weld in the process. Nobody was hurt in the process but better safety measures have been implemented since this incident. After wallowing around in sadness for a bit I spoke to my mentor, a consortium of engineers and my welder and slowly began piecing together the evidence for what had happened. The short answer was that I had been building more pressure than the transducer was reporting, by a factor of 3-5x. Huge error bars caused me, the operator, to believe to have been at a lower pressure than I really was, causing me to push the tank to rupture far beyond the intended testing pressure. While this is sometimes done in industry it was absolutely not what I had been wanting to do as it requires now I repair or replace the tank. One of the engineers I interviewed last semester, Eric Pillai from SpaceX shared some valuable insight into the weld engineering process and results. There was incomplete weld penetration (IP), and some porosity in the welds as well as some specific points where the weld was weaker than intended. This was brought up (nicely) with the welder and the location of the failure was at one of the tack points. Had this been a perfect weld, then the failure should have been in what is known as the heat-affected zone (HAZ) in that the material should have yielded in the weakened area underneath the weld and not in the weld itself. Heating up the aluminum to that high of a temperature during welding draws the temper on a part of the material adjacent to the weld, reducing the weld temper from say t-6 to t-4 which should have compromised before the weld. However, this is splitting hairs and in all fairness the weld was good enough to have passed, and the failure was on my part for not having calibrated my transducers properly. This is quite good news as it means I can rather confidently take the tank back to the same welder for repair. Another bit of good news is that the tank is repairable. At first I had thought the tank was scrap but rewelding it is indeed a viable option. Now I have valuable data on failure and burst pressure. Simply put this thing will likely burst at 900-1100psi and I will hopefully not be pushing them this far. Speaking with James Bauer, the welder, also yielded some improvements in bulkhead design where he suggested a different shaped bulkhead for better penetration. All things considered, it was a very valuable test.

I have been spending the last few weeks since the test preparing and completing my test stand for further testing. I have installed the final blast shield on the stand which now allows me to mount my engine but more importantly conduct hydro tests more safely. The probability of another failure of that caliber is near zero now that I know what red flags to look for. I also will install a camera so that should I get another failure it get recorded. Lastly I’m going to get a calibrated digital gauge to measure my transducers against for the most accurate readings. Just to make sure they are in the right ballpark. Also a note on the welding. That type of welding in industry is done almost entirely by robot. Aluminum is a very fickle material to weld and is very unforgiving. In welding pressure vessels this is especially important as any imperfections are punished by the operating fluid by a much higher rate. For example a fault in 1-2% of the weld would result in roughly a 10-20% increase in the amount of stress upon that one point. This is why in production environments, engineers are very picky about how the welds are done, and there is a more detailed inspection process which normally includes a visual inspection, a pen test and an x-ray test to ensure proper penetration. Unfortunately I had not performed any of those tests, for visual I  had no idea what to look for, and didn’t really care too much about cracks for a pen test and an x-ray test would have been very costly.