Back in the pack: A fine 12-year-old

  A message arrived to the inbox one day, from a fellow I'd seen on the Prius forums a few years back.  Over the years I had sort of lost touch with active participation there, but he and some others had been tracking a strange issue that seemed worth specifically reaching out to me on.  Evidently some first-generation battery ECUs were having nasty corrosion problems with the block-voltage monitor wiring, sometimes leading to small fires and cooked computers, and a similar issue had apparently also begun showing up in a few early second-generation cars as well.  He pointed me to the Priuschat thread on the subject, which was getting actively updated with incident reports as they came in.  The posts there from "Texas hybrid batteries" and "strawbrad" are of particular interest, showing how mysterious green crystalline corrosion appears in otherwise seemingly fine monitoring connectors and eventually leads to shorts, diagnostic codes and even a few meltdowns.  And always seeming to begin in a particular place in the connector.

This was obviously worthy of concern, as my '04 was right in the age group that could have the issues, so it was time to revisit the innards of my battery pack to check up on things.  While in there I took the opportunity to make some other inspections and improvements as well.

[Images here are linked to larger copies.]
The suspect voltage-sensing connector So I've got the end cover off the pack for maybe the third or fourth time in the car's life ... but now with a new purpose: to head off a disaster in the making, if caught soon enough.  The stated usual problem seems to start around pin 22, at the indicated corner.  Nothing appears externally amiss here, but some of the early-onset problem symptoms had only been deep inside the connector itself.  Getting to it and pulling it actually isn't easy, as it's clipped down to a bracket on the bottom of the box, so pulling the ECU seems necessary just to get the connector up and out [and I want to have a look inside the ECU anyway].

My pins are pristine! To my profound relief, the problematic pins appear to be pristine.  No discoloration or crystalline stuff at all, just shiny metal with intact-looking plating.  Whew ... that's great, but doesn't solve the larger mystery.

The battery monitor wires are brought to the connector in a way to minimize the voltage gradient between any two pins -- successive taps start at one end and alternate across the two rows, working along the connector, so that the extreme positive and negative ends are far apart and no more than about 35 volts should appear between any two adjacent pins.  Surely the connector housing can deal with that, so the corrosion is not likely due to some conductive fault of the plastic.

Back-side wiring is okay too The wires going into the back of the connector look fine close up; no detectable green crust on the crimp lugs, as seen on some other examples.

Note the unpopulated positions in the harness connector -- the ECU and its interface appear to be designed for a larger pack with more taps, such as in a Gen-1.  Toyota was leaving their options open here, I imagine.

Nothing amiss inside the ECU With the ECU removed and its cover popped off, the pins into the monitor board can be inspected. No hint of anything going on here either, around pin 22 or any of the others.

Possible small contact pit in pin 10 There's a very small dark area on pin 10, that looks a little like the point-of-contact pits commonly found on such connectors when they stay plugged in and never moved for a long time.  Pins of this type *should* be a sliding, self-cleaning type of contact, but only if someone comes along to unplug and replug them once in a while!  This isn't a pit after all; whatever the stuff is scrapes right off by hand and leaves the clean pin behind.

Connector lug is clean A couple of the wire lugs are backed out for inspection too, and even under decent magnification there's still nothing notable or vaguely suspicious.

  So the plot thickens.  This car has been on the road longer than a lot of second-gens under a reasonable variety of ambient conditions, including various roadtrips around the country to climates other than New England, although I can't say it sat soaking in high humidity or any other obvious persistent condition all the time.  Electrolytic effects in different environments?  You wouldn't think that appreciable current passes through any of these pins, as they're *voltage* monitors.  And there shouldn't be any corrosive vapors drifting over from the battery itself, as the modules are sealed and the chemistry is fairly benign.  For the moment, I'll just accept having dodged this particular bullet.

White fuzz on my nuts As long as I'm this far in, it makes sense to pull the main cover and check out the module interconnections I took note of six years before.  Minor deposits had started to appear on some of the terminals back then and I'd decided it wasn't worth worrying about at the time, but it got me thinking that I might want to eventually pull this apart and clean things up some. 

The white fuzz has clearly grown quite a bit in the interim.  Now, having crusty white fuzz on your nuts is never a good idea, so it looks like now would be a good time to get to this.  And I've got just the right materials for the job.

    A practical Prius history lesson

  Back in 2010 I was hangin' with my fellow hybrid geeks at Hybridfest in Wisconsin, where Dave Taylor had a booth for his startup battery rebuild shop called ReInvolt.  He began by rebuilding failed first-generation Prius packs from the more robust and newer second-generation modules, giving new life to the beloved rides of some of the early adopters.  The business flourished and grew to cover many other types of hybrid vehicles, and eventually became an arm of Dorman Products, and Dave's still involved in the field.  Anyway, we were chatting at the Fest and I mentioned the deposits I'd found on my battery terminals, so he handed me a little bag of his latest innovation -- new tin-plated copper straps, designed to reduce dissimilar-metal issues at the interconnect points.  Given that no immediate action was needed on my pack, I squirreled them away in some box back home and almost forgot about them, until now.

This is a chart mostly aimed at the construction industry, but is useful here too.  Slap two different metals together in the presence of an uncontrolled atmosphere and moisture, and the degree of interaction can be roughly predicted by how far apart they are on the scale.  Add electricity, and things can get a lot worse!  What's most interesting about the white fuzz is that it's clearly heavier on the *negative* battery terminals, even though both studs and the copper are uniform.  Given that both charging and discharging always involve a little loss as heat, one can assume that more energy flows electrically *into* the pack than comes back out, and thus electron current spends more time moving from the copper to the steel of the negative module studs than the positive ones.  I have no idea what the reaction might be or what the white stuff is composed of -- it's sort of flakey and amorphous, not really sharply crystalline.

The safest bet is that if metals are going to touch each other long-term, they should be as close as practically possible on the galvanic scale which the tin plating accomplishes.  I don't know what process is used to plate the straps, but there are any number of simple-to-use plating solutions for the electronics industry.

Starting to remove interconnect straps First step is to brush off most of the fluff with everything still in place and vacuum up the detritus, and then start careful disassembly.  My continual paranoia is that a nut will be corroded onto its stud so hard that the stud will just shear off, but everything seems to be loosening up quite smoothly and the studs are mostly clean underneath.

These straps are often referred to as "bus bars", including in the Toyota manuals, but that's really a misnomer as each one only connects two points serially rather than distributing over multiple parallel points and thus don't form a "bus" at all.

Sensing tap lugs pop out bottom-first On the front side of the pack, I have to also deal with the cell-balance sensing wires that run to every two modules.  They pop out easily bottom-first, by pushing on the crimp part through a small hole conveniently behind it and then lifting out.  The flat lugs for these are still clean and in good shape, though, and the fuzz doesn't seem to have migrated to them at all.  Their base metal is probably copper, so that kinda makes sense.

All interconnect parts removed The copper has become very dark over time and also has minor bits of green stuff here and there, which is what we'd expect from copper.  If it was up on a roof, it would look great!  There's still plenty of shiny "meat" left under where the nuts and stud bases make contact, so these probably could have gone a lot farther without incident.  The nut-fuzz appears to *only* be surface deposit, e.g. nothing has actually eaten into the metal of the studs or nuts and I can gently scrape the stuff off and get right down to the original steel surfaces.

If Toyota was thinking about the metal interactions here at all, they probably planned for an approximately ten-year safe lifetime and didn't bother treating the copper straps.  These are 12 years in, and probably right on track with whatever degradation curve they might have come up with.

Examining the sensor crimp connections The voltage-sense wires look generally okay, but a couple of the lug crimps have a touch of corrosion and darkening inside.  A brief attempt to get a little solder into some of these, heavily fluxing to try and clean the oxides out first, meets with dubious success.  Since they're all very low current connections and otherwise nominally sound, I give up on the idea of refurbing them all.

Nice clean parts to go back on The original nuts spend a while getting shaken around in paint thinner and wiped off, and seem fine once the fluff is gone.  I've actually got quite a few extras, part of a big batch of batteries and wiring parts I sort of inherited after one of old hands from the Prius_Technical_Stuff yahoogroup passed away in '07 or so.  He had been doing some of the earliest work on adding extra packs to the Prius, had an accumulated stack of salvage parts to build from, and his widow had no idea what to do with any of it so I stopped through one winter and wound up bringing some of it home to help clear her storage.  Since the original nuts are in fine shape I've saved out only four of them as display pieces, and swapped in four clean ones from the extra stock.  They'll go on the four critical points of positive, negative, and the safety-plug connections.  Everything gets a final wipedown with a little DeOxit before assembly.

The batch of straps from Dave is evidently for a Gen-1 pack, with several left over after determining that I'd need 26 of them here.

New straps installed Ooh, shiny!  Everything fits back together beautifully, and as I'm torquing the nuts down I can feel how they're designed.  The slightly conical attached washers are intended to form a spring-loaded connection against the strap and keep everything tight to the stud base, so there's a span of rotation between initial contact and "tight" where that's obviously happening.  This helps mitigate a common problem seen in other electrical wiring, that being that soft copper creeps under pressure.  In electrical boxes, for example, I've gotten into the habit of bringing a screw down tight and then just holding the static torque on it for maybe ten seconds, and it always moves just a little further.  Even here, a second pass along the nuts after a few minutes to snug them all down uniformly has some of them turning just a wee bit more.  Still being paranoid about stud shear strength, and not pushing it on the final torque.

Fuse plug is perfectly okay The safety plug has no signs of any issues, even though it undergoes the same heavy flows of current.  They must have picked just the right plating for those bolts.

Pack heat profile Once things are together and functional again, it's time to go out for a little charge/discharge abuse-testing and then look for hot spots.  If anything, the interconnect straps are the coldest part of everything.  More warmth is seen down in the bottom of the cells, which may be slightly influenced by heat coming up through the floor of the car -- note the hint of a warm area toward the right where the exhaust pipe passes underneath, but if anything that's a minor factor.

    Finished but not forgotten

  It was nice to have closure between those early discussions with Dave and some other battery-rebuild enthusiasts and finally getting this refurb done.  The whole job was a little tedious but quite simple, and I was done with it well before I thought I would be and thus the car was *not* going to be down the entire weekend after all.  Meanwhile, an acquaintance had been poking me to join him and some other hybrid folks at the big Nantasket car show, which was happening all of a day after I wrapped up the strap swap.  So what the heck, I thought, why bother putting the car back together when this could be a great geeky display of things most people never see?  After printing up up a couple of explanatory signs, I was good to go.

At Nantasket Beach car show That car show was a much bigger deal than I thought it would be -- must have been a half-mile of muscle cars all lined up along the beachfront road, and here I was right in the midst of it with my 12-year-old original battery exposed to the masses.  And people were interested.  Not since the fuel spike of '08 dropped off did I have the opportunity to stand out in the hot sun all day handing out flyers and knowledge -- it was fun to be back in that event groove for a while, even if most folks go to shows like that to ogle all the "old iron" and street stompers.  Like I reflected on in another article, I totally understand the draw of the classics and it's still fun to contrast all the different technologies and their evolution.

Testing monitor tap currents The covers stayed off the pack for another week-plus afterward, to also show off at a local cruise-in and then a social event the following weekend.  Meanwhile, that "pin 22" problem was still nagging me a little.  One morning the inspiration to just go *measure* the currents in some of those sense wires struck me, and rather than mess with the connector it was easier to just pop the sense taps off the interconnects and drop a milliammeter in series with selected ones -- especially the wire to pin 22 which comes from the negative end of the pack on the left here.

The measured currents certainly aren't zero, and sort of pulse up and down slowly when the car is powered on.  I'm on the 2 mA scale here -- while the levels are only a few microamps, what I'm measuring never goes *negative*. e.g. it always flows, however modestly and however intermittently, in a consistent direction between the battery and the ECU.

  What's especially interesting is that the lead to pin 22, from the negative end of the pack, carries a little more current than the other taps, and as I thought about this more one thing suddenly became obvious.  The ECU is a fancy voltmeter, and as such needs a ground reference.  This would logically come from the negative end of the pack, even though the entire battery system is isolated from the *car* -- the point is that as each tap is measured all the way up the pack, a small current must get drawn between it and pin 22.  This would make more far more sense from a design and circuitry standpoint than trying to only measure deltas between adjacent taps.

The slow pulsing nature of the current is also interesting, as it implies a periodic cycle of measurement with a set of samples taken every so often.  With the car still powered up I waited until the current peaked and went back to 0 and then disconnected my meter, leaving the negative-point tap floating free.  Right about the time I expected the next pulse, the car went "bip!" and lit the Red Triangle of Doom, and the battery *fan* came on at some intermediate level -- an informal confirmation that a break in a sense tap isn't detected instantly, but only during a measurement cycle.  The error code on the scantool was P3030, "disconnection between battery and ECU", and a quick wander through Toyota's description made the likely function of pin 22 carrying "GBB0" even more obvious.

I reflected the details on these findings back to the fellow who had originally given me the heads-up on the problem:

So I've got my pack all strapped back up with happy tin-plated bars, and
have been running around with the pack naked to the world in the back
because it made a great demo at a car show last weekend.  "Naked prius"
revisited, in a way, which nicely bookends the whole decade...  anyway,
in the process of pulling together stuff for the inevitable webpage,
I thought hey, why don't I go out and *measure* some of those sense
currents and see if there's anything funny about pin 22?  ... well,
rather than messing with the orange connector it's easy enough to pull
the tap lugs off the pack and just run 'em through a milliammeter on
their way into the bundle.  And I found something interesting.

Sampling currents are on the order of microamps, but what happens at pin
22 is that it sees pulses of up to 25 uA about every 4 seconds.  Inserting
into the path of another tap farther "up" the pack [in terms of voltage]
shows similar slow pulse behavior, but not as high a current -- maybe
4 or 5 uA, and bounces around a little more.

So here's what I think might be happening.  Pin 22 connects to GBB0,
the negative end of the pack, and as the block-scanning happens, all the
other tap point voltages are *referenced off it*.  I.e. the tests are
likely from a given tap to its effective ground, rather than taking
each individual delta between adjacent ones.  So pin 22 *does* get a
tiny bit more abuse over the long haul, thus it's "special" in that way.

It may be that pulling and reseating that connector a couple of times
earlier in a given car's life might head off the whole thing, who knows.
*Something* must have put stray resistance across that joint in some of
the ECUs as the cars were built, or the choice of metals right there
wasn't right, but that wouldn't explain how only some grow bad stuff.
On the other hand, the "white fuzz" on the bus-bar nuts was quite a bit
more than six years ago, so even though that's a much higher-current
scenario maybe *any* potential across galvanically disparate enough
metals is eventually going to have its effect.  "Texashybrid" probably
sees a lot more ambient humidity than I do in general.
What would likely be most instructive is the complete history of any car whose ECU sprouted the problem, including climate, garage-kept vs. outside, whether the owner generally drove windows-up or windows-down, A/C usage, and any sources of humidity commonly in the car like plants or pets.  Comparison with the amount of deposit on the pack interconnects would also be useful.  Without knowing all the circumstantial differences between a car with the problem and one without, anything further is probably just speculation, so heck, here's a little more just to throw it out there.

We know that to measure a system you must disturb it, however slightly, and a voltmeter has to draw some tiny amount of current through a completed circuit to move a meter or charge a sample point.  Others observed that the series resistors going into the ECU's measurement chips are only 1K, so what if there's actually a very brief inrush of higher current as a given tap is switched in for sampling?  That could lead to spikes of a hundred or more milliamps for a very short time, but it's still unclear if even that would be relevant.  Both the pins and lugs appear to be gold-flashed at the contact surfaces, in theory lessening the dissimilar-metal issue.  So while I think I'm on to the reason pin 22 specifically has been the "first to go" in most instances, the question remains, what's growing those crystals in the first place??  Maybe somebody should get in touch with those guys on the planet Krypton, since they're the recognized authorities on that sort of thing.

_H*   160615