
Real I.T. Horror stories submitted by other people. Submit your own and WIN ::
Stories arranged by tag: Emergencies ::view all
Ian Lyons
Title: We have been taking the tapes off site - right?
|
In a previous role, the office was in the 18th floor of the World Trade Centre in Moscow. There weren't many skyscapers in Moscow back in '93 - and ours provided the best vantage point for snipers shooting down at the Russian White House during the coup. Shooting back up was the militia with big guns and RPGs. The top 2 floors were completely blown out. Our floor was peppered with 20c coin sized holes in the windows.
All the staff evacuated safely however the IT manager was looking ghostly pale - he'd realised that all the tapes were sitting in a neat pile in the server room. He decided that he had retrieve them - walking up all 18 floors, emerging into live fire, crawling on his stomach grabbing the tapes and fleeing for his life.
The next week we instituted an offsite backup rotation. Good thing too, because we had a massive failure later that year when the dry ice failed to keep the server room cool enough - but that's another story ...
|
Read this story::
Rodney Vandermeer
Title: Disaster Revovery Gone Wrong
|
1) Critical Line of Business System crashes - Files system (FS) unavailable as the storage it is located on has failed and Prod FS is not recoverable.
2) Recover system Option 1; Remap copy of Filesystem at DR to Prod system Option 2: Create a new filesystem and map to Prod system (create storage volume and connect to app server). Copy data from DR Option 3: Create new Volume and restore from Tape. Will take a long time (~ 400GB of small files)
3) Problem with Recovery Option 1: Robocopy (host based copy procedure) has not worked for months? Option 2: Same problem as Option 1 Option 3: Ok, but will take overnight and next business day to resolve - massive outage
|
Read this story::
mark hide
Title: The Groceries shops dont understand computers :)
|
I was only 15 at the time (1997) and the place i was working at was a local supermaket store which was a job then back when i was in school. i was heavily into computers and just learning c+ and other ms linux opps, the store check out system was a clear hit before any coles or Woolworth stores had ever imagined but.. the story goes on.. the operating system was the dinosaur at the time running a dos 3.30 with a interface that was ahead of time then but as the server, that was a mistake!!! the check out systems used to run 386 2mb ram and a 20mb hard drive (wha!?!?!?!) it was reliable but the software running was overwhelmed on overload, with a barcode system running over a 1400bps .014kb of a twisted wire which ran up in the celing to the back office. the boss got to know me as a teck guy and know all... so he asked me to figure why at the end of a day or when purchasing an item the computers were slowing down and freezing the checkout systems (mind you he would do the purchasing when it was peak time in the afternoon) i did not know at yet of the system until he asked me... so i went to investigate... mind you i was a kinda nerd then and the girls were a total distraction :D but i found out that the network was based on a 1983 modem switch network and three check out systems were using the single .014kb system. when the staff were serving customers with scanning the items the data was being sent to the back office every hit, the boss the other hand was doing purchases via a dail up modem (the ones where you place a speaker (sender) to the reciever and a mic (reciever) and that was chugging on the same network but a little faster (1.2kb) i told him to wait either before or afterday so it would not collide with overloading the system. it worked the next day... the next thing was even funnier but that might be another storey..... :) thanks for reading!!!!@@@@
|
Read this story::
Adrian Hendon
Title: Coffee and server rooms don't mix
|
Our servers got covered in coffee one evening whilst our IT manager was doing the daily back up - he tripped and tipped his whole mug onto the servers which then caused a fire and we lost crucial financial and customer information. The damage cost us $1.2 million. Drinks of any sort have now been banned from the IT room and our IT Manager was made to wear a hat shaped like a coffee cup for a week..
|
Read this story::
Milan Tomic
Title: The Hero takes a fall
|
As a young entreprenerial go-getter, I established my first computer networking business in 1995. Out to impress, I would attend all my business meetings and call-outs in a suit and tie.
One morning, I was contacted by a client with an emergency. The main server was unable to boot and the client was unable to process orders.
Clever person that I am, I had set them up with a RAID solution that meant that all their data was backed up on a second drive. So I went out to site, rebuilt the OS and switched to the backup drive and they were back in business.
Before closing the chassis of the server, I removed the faulty drive and asked the boss to get a few people to test the system.
The users were back up and running with no data loss.
The Boss was over the moon, lauding my every move and my incredible confidence and ability at such a young age.
The Boss left the "server room" and asked his staff to start processing orders. I screwed the lid back onto the chassis thinking of the fat cheque I was going to get for the emergency call out.. smile from ear to ear.
That smile immediately disappeared as I turned to grab my tools. I froze, choking as I realized I had screwed my tie into the chassis and as I turned it went taut, bringing the server crashing from the middle of the "rack" a good four feet to the concrete floor.
The boss walked in.. my guess is to tell me that the server was down - only to find me bucked over the destroyed chassis being choked by it.. unable to remove my tie or extract it from the chassis.
I folded the company after that.. it cost me much more than I made on the emergency call. True Story.
|
Read this story::