Well, work has been interesting in the past 50 hrs or so. At approximately 2:50pm on Friday, the main power transformer serving campus failed. If that wasn’t exciting enough, some of our battery packs for UPSes in our server room started venting gas (of the hydrogen sulfide variety). I’m not sure what percentage of our servers got shut down gracefully but I think at least a few went down hard. Six IT staff went to the hospital via ambulance for treatment for gas exposure. Power was restored late yesterday afternoon. If I understand correctly, it’s a temporary replacement transformer so there will probably be another outage … hopefully planned. Last night through this morning was spent getting batteries recharged and back online along with the network hardware (= switches). At noon there was a meeting which I decided to attend for information (I wasn’t involved in anything to this point). Ended up getting roped into working (most of which was acting as a “buddy” in one of our server rooms with battery packs). Most services were restored by 8:30 tonight though a few are still being worked on. Hopefully those aren’t too difficult for the staff remaining to restore. Tomorrow of course I’ll probably get to deal with fallout from office workstations that went down hard. Yay.
journaling: enabled
it's a geek's life
© 2024 journaling: enabled is Proudly Powered By WordPress | Theme by The Cloisters
Wow! Were the power failure & venting gas issues directly related? (As in, did the power failure cause the gas venting?)
You definitely win the most exciting work story trophy* for a while. It’s going to take a lot to top that too.
Hope everyone is ok (you said they went to the hospital, not what the outcome of the visit was) and that most of the workstations come back ok. Except the ones that you really wanted to get rid of anyway…those, hopefully, lost their magic smoke and will now make excellent doorstops
*Disclaimer: trophy does not actually exist. If it did, it would, no doubt, be spectacular and awe-inspiring.
The gas venting didn’t occur until the the power load was on the batteries (after power failure), but since the battery packs and continued to function the cause is unknown. We actually have APC techs out on-site this morning to look into it. Haven’t heard much in the line of office workstation issues thankfully.