January 6 – 7, 2018 —
Saturday afternoon, three days after arriving back at the boat, and one day after the non-event storm, I decide to take a long bike ride for exercise and pick up groceries to start restocking the galley. I was gone for over two hours but not more than three. We often leave the boat powered-on unattended for far more time than that.
Upon returning to the boat, laden down with supplies and fresh vegetables, I threw open the slider and was greeted by that stale, old-boat odor, everyone has probably experienced. Funny, I hadn’t noticed it when I first returned to the boat from being away for two months. Perhaps I stirred up some foul odors from the bilge, or maybe it was because I had just used the holding tank. It was the odor I know all too well from restoring an old boat: a combination of wood rot, mildew, toss in a little bilge nastiness and some diesel drippings, and there you have it. It is not an especially distasteful odor, but when you experience it, it is unforgettable and reminds you that you are likely on an old, usually poorly maintained, boat. An odor that has no place on Quintessa. I was obsessed with finding the source.
Sticking my nose in every locker and cabinet I was without a clue—even the bilge smelled sweet. As I worked my way aft, I thought it might be a little stronger. Let’s check every drawer and cabinet. The first drawer I opened, which happened to be my sock drawer, knocked my sniffer on its butt. OMG, did some rotten eggs fall out of the garbage which happens to be nearby?
Well that instantaneous thought was just my knee-jerk reaction. I knew exactly what it was, and what the source was—and it wasn´t my socks! It was hydrogen sulfide, and the only place it could be coming from was a battery. One of the three big house batteries is located behind the chest of drawers in the aft cabin. 
Why would this be smelling? Do we have a problem with the battery? I threw back the mattress and opened the quadrant locker where the battery is located. The smell was now overwhelming. I placed my hand on top of the fiberglass battery box and jerked it back—the box is boiling hot! Holy shit, a runaway battery. What do I do?
I was not entirely sure what to do. I have never had a runaway battery before but I have read plenty of scary stories. I know at the very least I need to cut the power to the battery, and ideally, isolate the battery from the rest of the bank. I rush to the nav station and throw off the charger and the inverter. That was easy, but how do I isolate the battery? I know I can turn off the master breaker, but it is now after dark and plunging the boat into total darkness would not help my continuing efforts. I know from having the electrical system completely rebuilt, there is a sequence of switches to transfer the load to the bow battery that normally runs the thruster and windlass. Where is the schematic? It must be in the emergency folder we keep right by the nav. station. Not there.
Time is wasting, I don’t know whether I have seconds or minutes before the battery bursts into flames. In a panic my thinking is not clearly logical. This is why they say we should have standard procedures and drills. Reactions must be instantaneous. I need a second to think, calm down, I tell myself. Find the schematic. Coincidentally, I just had the DC schematic reprinted while I was back in the states. Forget the old copy, a new one is in my backpack. Quick, flip it open. Oh yes, one electric solenoid and one breaker and I can transfer the load and shut down the bank without an interruption. It works!
That is good for the moment, isolating the battery from the rest of the bank will take longer. My guess it that three minutes have elapsed. I need to see what the battery is doing. I grab the infrared thermometer and head to the aft cabin. I aim the red dot: 170 degrees. Wow. I can see the gas seeping out the safety vents on top of the battery. The sulfuric acid is being boiled off. The battery case is already starting to distort from the heat. I don’t know what the ignition point is but if only the temp would start going down I could breathe a little easier. I need to monitor it and have plan “B” ready to go. Get the Halotron extinguisher ready, get the fire blanket.
Now I need to think about sparks; hydrogen gas is highly explosive. I flip off the hot water heater and outlets. Next ventilate the boat: I throw open the aft hatch but before I flip on the bilge blower, I dwell for a nanosecond, yes, the bilge blower should be ignition-proof, I am sure it is. OK. Phew!
At least five minutes have gone by, time to re-check the temperature: 175 degrees. Holy shit, it is not going down. What if I need to get the battery off the boat? It is an 8D battery, 155 pounds. There is no way I can lift it up out of the compartment by myself, even if it wasn’t burning hot and I could use the handles to lift it. 
Lifting the whole battery box with the battery would be even more difficult, even with another person. But, there isn’t even anyone else around. Wait, that won’t work anyway, the battery boxes are screwed down from within. Should I call the boat caretaker? Even if he is at home, I don’t know that he could get here in time.
I don’t know what to do and I don’t know how much time I have. What if the temp continues to rise? Maybe I have time to check the internet. I need answers. Thank you, Google. A temperature of 170 degrees on the case is likely 300 degrees inside and the case will be getting soft. At 200 degrees the case will be failing and at 250 degrees it bursts into flames. This scenario assumes there is not a hydrogen explosion! According to these sources, at 170 degrees, the battery has started to runaway (an unstoppable self-perpetuating reaction) but can be arrested if only one cell has failed (a 12-volt battery has 6 cells). If the failing cell has breached into the next cell, the runaway is a certainty, and containing the fire is the only option. Firefighters have special metal boxes to enclose a runaway battery until it burns itself out. This is basically a Hoverboard or Galaxy Phone scenario times 1,000, only with slightly more stable AGM batteries as opposed to Lithium-Ion.
There are no good options. Getting the battery isolated is probably the next best bad option! Back to the schematic; three 400-amp fuses, but which one isolates battery no. 3? I now discover there was no labeling on the fuses (my fault), and the schematic plan, not being a wiring diagram, does not identify which fuses are where (another shortcoming). Minutes tick by but eventually I figure out which fuse is which, and attempt to remove it.
Removing the fuse presents the next set of challenges: The fuse was installed with a socket wrench with a long extension in ideal conditions. To dig out my full tool kit with all the socket wrenches would take at least 5 minutes—time I don’t have. Anyone should be able to remove a fuse in a crisis with basic tools in less than a minute! 
Maneuvering with the only wrench I had handy, a box wrench, in a tight space meant I needed to work the wrench at bad angle which was rounding off the edges on the nut and not budging it. Further complicated by the fact the wrench was too long to swing it more than a few degrees. On top of that, the wrench was not insulated and I was working next to a large thru-hull valve which, of course, is grounded. One slip or misalignment would result in an electrical arc with the full amperage of the battery. One big spark! It would probably instantaneously weld the wrench to the thru-hull, and momentarily blind the person on the other end of the wrench—me! My quick solution was to pull the next fuse down the line (isolating two of the three batteries) then go after the correct fuse with a pair of Vicegrips. Not pretty or safe, but it got the job done.
After isolating the battery, the temperature stabilized and slowly started down. Huge relief! I baby-sat the battery for the rest of the night and ventilated the area for over a day. The next morning the temperature was only down to 140 degrees, and in 24 hours it was still warm but just below 100 degrees and dropping.
In conclusion, after having time to do some research, the steps I took intuitively, were all correct. Isolating the battery from the rest of the bank may or may not have made a difference in arresting the runaway. I figured it would have at least saved the other two batteries from being pulled down by the runaway. Unfortunately, as is usually the case, that damage was already done when the first battery first failed. As it turns out, all the batteries were at the end of their service life (five years and over 1000 cycles) and it is always prudent to replace all the batteries in a bank even if only one is bad. The only issue we will have now is having to run on a single 8D in the other bank (bow battery) until we can replace the three bad ones this spring. No powered winches or microwaved meals.
With the benefit of further research, I can say that batteries can get into a runaway situation from either internal causes or external causes. What likely happened in my case was the batteries were failing (we had planned to replace them this year anyway) and with recent heavy use of living aboard in the winter with the heat pumps running off the inverter one of the cells may have developed a slight problem causing it to not take a charge. The AC charger still pumps out the programmed amperage but instead of the power going into six cells, the same amount of power is now going into five cells in that battery. This causes a slight rise in temperature of the cells which reduces internal resistance of the battery and tricks the automatic charger into thinking the battery can take more amperage than it really can. The charger automatically increases its output which causes the battery to heat up further, causing the charger to output even more, and so on and so on. Thus, creating a negative feedback loop, perpetuating the problem. The battery gets hotter and hotter until another cell fails and then another, at which point it is an unstoppable, runaway reaction resulting in a fire or explosion.
The chemical energy in a relatively small volume such as 245 Amp-Hour, 8D battery is downright scary! The energy stored in one fully charged 8D battery is roughly equivalent to 1/10th of a gallon of gasoline, and in case you are wondering, it can probably be unleased almost as fast! Fortunately, this is not a common occurrence with AGM batteries. Usually they just get weak and die peacefully. For now, I think I will steer clear of Lithium-Ion replacements, which are more powerful and even less stable, until the bugs get worked out.
One final note and one final lesson: Had the temperature sensor on the failed battery been operating as it should have (a known problem I have now addressed) the charger would have detected an excessive temperature rise (more than 20 degrees F over ambient) and reduced the power before anything got out of hand. Regarding the lesson learned: 
Play out the what-ifs. Hind sight is 20/20 but if one has to remove a fuse in a hurry, can it be done in an emergency, under less than ideal conditions, with ad-hoc tools. 
I now have a specific-purpose insulated hex wrench attached in easy reach right next to the main battery fuses. Then what about having clear information on what is what? I also now have a wiring diagram of the fuse and battery wiring in addition to the schematic. I am not about to redesign these critical systems on the boat but perhaps there should be more emphasis on engineering-for-service in the design and building process. Installing a system in a boat is one thing, servicing the same system can be an entirely different procedure. Boat builders take note.