Getting wet on the job.
Thanks to a story on theregister.com (On-call story) this morning, I got thinking about a call out I did almost 6 years ago which was messy as hell, could have gone badly wrong, and did start me deciding to switch jobs (Mostly due to the now Husband pointing out the stupid risks that were taken).
Its October 2011, and I'm heading back home after a long day out on site with a client. The radio news in the service van has loads of weather warnings about rain and spot flooding due to a freak weather system in my city – and as I get closer, yep, it is bad. Visibility is maybe 50 metres, so I'm crawling along and eventually pull off at a fuel station to figure out wtf to do, and call a mate asking if I could crash on his couch for the night (It was looking safer than driving around the city to get home)
That’s when I get two phone calls – one after the other. First one is the IT manager for a hotel group (our biggest client at the time) letting me know one of their main hotels, located about 150 metres from the river, is beginning to flood (In a fit of building wisdom, the staff offices and all the IT stuff were located 2 levels down in the basement). We have a chat, he's unable to get there (I found out afterwards his wife flatly told him he wasn't to risk it). I let him know where I am, and that as it is, I'm planning on crashing on a mates couch tonight as I don't think I'll make it home. Discussion over, he understands, we hang up, and I go finish the hot drink I'm after buying.
Second phone call is about 5 minutes later – and it's my Boss. No asking if I'm OK or other pleasantries – he directs me to go to the flooding hotel, and pull the servers out. The tone is the "I'm the Boss and giving the orders here" tone. I do tell him no – I'm not convinced I'll even make it there, it’s hammering down rain, and the police are advising people to get home and shelter up – get off the roads and stop driving in other words.
His response is simple – go there, or don't bother showing up for work tomorrow. Not his problem, it's mine, to save the hotel. The call was bad tempered to start, and ends the same way. To reinforce it, he sends me a text message (which also contains a few swear words).
I start driving – I'll skip this bit, let’s say that I had to take several diverts, back out of water twice, saw several crashes and flooded cars and houses, and what should have been 30 minutes took almost 90 minutes. With two other phone calls from the Boss on the way of a similar sort to the first. Needless to say, I'm not in a great mood when I get there.
Park up outside the Hotel and head in – after talking my way past a security guard, and I find the Hotel General Manager having the worst night of his professional career. There is no power in the hotel, the basement is flooding, the fire brigade want the place evacuated (He was holding his ground on that request, as moving the guests would be near impossible – it was still blasting down rain). But he's the dedicated professional – and really glad to see me. A hot drink is brought, and the Maintenance manager summoned to update me.
It's not good.
The main problem is the dual-redundant, failure proof pumps installed in the basement to counter this, have failed. And no hope of starting them. The basement level is flooding – slowly, but getting higher. There's no power, so no lights down there except for the emergency lights (Local law, thankfully, required 6 hour emergency batteries in the lights). He evacuated all staff from the offices on the same level as the Server/IT room – so he has no idea what the exact water level is. The level underneath that (three levels down) is already totally underwater.
He gives me two guys to help (thankfully, both look like they pump serious iron) and tells me I'm nuts, but good luck. I grab a few screwdrivers and my head torch from the van, drop my phone (I've ignored another phone call from the Boss) on the driver's seat, and we head on down.
Bottom of the stairs, and the water is about 4 inches high – just enough to flood the work boots of course. It's filthy water as well, adding to the fun. We open the server room door, and I start at the bottom – unbolt the POS server, it goes up the stairs. Return for the next item..
It's then I realise the phone system emergency batteries are behind us in another rack – a nice, compact block of 8 or so car-like batteries. In a cage I can't open to disconnect them. With exposed terminals. And the water is creeping higher. When we entered the room, it was just over my work boots, now it's half way up my shin.
I warn the others, stay away from that cage, and we work flat out – I unplug the UPS and pull the battery isolation connector on the back – too heavy, don't want to waste time on it. Main AD server, remote access server get unbolted and moved. Switches – water is now just above my knees.
Myself and the two guys keep at it – anything we can unscrew and move from the server room we unscrew and move – CCTV, POS interfaces, all the things that make up the backbone of a modern 4-star hotel and its systems. All unbolted, carefully kept above the water, hauled out of the room by torchlight, up the stairs to a holding room on the first floor three levels up (I suspected the ground floor might be wet before the night was out).
In the end – two hours later, with the exception of the phone system cage (which was locked) the server room looks like a vandal went into it with a crowbar, everything ripped out. Cables float in the water like straw. Said water level is now up to my chest and about to hit the terminals of the phone systems battery pack, so that's it. Extra Omnes – everyone out.
We meet two fire brigade guys coming down the stairs as we head up – they were going to order us out. As we get out of the water on the staircase, there comes a distinct frying sound from the server room, and a smell, as the battery pack short out from the filthy river water reaching the terminals.
On the surface, it's now well into the small hours of the morning. A fire brigade officer tries to chew me out for being an idiot, but I'm tired, soaked, cold from the water and sweating from the exertion at the same time. Water is pooling around me where I stand. He gives up when he sees I'm beyond caring, and leaves me alone at the quiet word of the Hotel General Manager.
A fire brigade medic asks a few questions, gives us a once over, says no damage he can see, but we need to be decontaminated due to the water. Simple way to do it – strip off, and a low pressure freezing cold hose plays over us. The fire brigade give us 'emergency clothes' – basically something like a tracksuit pants and hoodie, thin but warm. The existing clothes are dumped into plastic bags, and never seen again.
Hot soup is poured into bowls for us, and I'm flatly told I'm not going anywhere till I warm up and eat. I eat.
Feeling a bit better, I head back to the van. The same fire brigade officer asking me questions earlier comes over again – asks why did I do it. I show him my phone. He notes a few company details from the side of the service van and tells me safe home – and the best route to head for. The flooding is already dropping, so the drive home was routine apart from me being distinctly able to smell myself.
Get home – the husband is NOT impressed. quick explanation, Super hot long shower, and I crash into bed. Before I wake up after midday, my phone will rack up many missed calls from the Boss.
The aftermath is swift.
The Boss gets two phone calls he probably regretted – one from the husband (I should mention, he worked as a professional Health & Safety type at the time), who personally, and then professionally, rips into him. I found this out afterwards, as I was still asleep when the call was made. The guys in the office tell me he was the colour of a sheet of paper by the time that call was finished.
The second phone call is from the Fire Brigade – following up to see if I was certified for working is water, flood hazards, confined spaces etc. The boss has to answer no – resulting in another fun phone call for him. And a full health and safety audit for the company shortly thereafter(it failed, spectacularly)
For those who are wondering how the pumps failed – they were never installed right, and never had been tested under flood conditions. Also, the control panel was not waterproofed, and was among the first things to be submerged in flood water.
And finally, after about a year of steadily worsening relations with the Boss (and yeah, I suspect this was one of if not the main reason) I left. Discovered afterwards they lost a bunch of clients as a result.
And last time I was in that Hotel meeting mates, I was still given a free drink. Same General Manager.