A lady half my size was six inches from my face screaming like a drill sergeant – “DID YOU REALLY DELETE OUR DATA?!” Her anger, emotion and frustration were boiling over, and I her sole target.
“ARE YOU THE MOST STUPID PERSON IN THE WORLD!?”
….
Somethings you remember exactly where you were in a particular moment. These flashbulb memories stand out in our lives and can be recalled in detail even though they happened long ago.
9/11, Challenger explosion, fall of the Berlin wall, moon landing, JFK assassination.
These moments can also be personal in nature; a big event that only affected you. Things like marriage, a child being born or a particularly good or bad day in your history that sticks.
We all have a number of these memories that re-surface from time to time and, even though they occurred long ago, still affect how we go about our lives. One of these days that is etched upon my mind occurred in 1982 in a small town outside of Kansas City where a lady half my size got up in my face like Marine drill sergeant and yelled:
“JUST HOW STUPID ARE YOU”? “HOW CAN ANYONE BE AS STUPID AS YOU ARE?!”
This line of questioning went on for what seemed like forever, but in reality, it was probably less than a minute. It did, however, change how I approached something so valuable as data for the rest of my career.
Let me provide a little background. I was working for DEC (Digital Equipment Corp), which at the time was the second largest computer company in the world. I was a systems engineer which meant that I fixed computer hardware. In today’s world, it can be weird to think that computer hardware repair was an entire sector of the industry, but back then it brought in more revenues than new sales did annually. Most of us understand that computer systems looked very different in those days – think several refrigerator sized cabinets in a row with disks that were each the size of washing machines.
 |  |  |
The DEC PDP 11 series of systems were incredibly popular in the late 70’s and 80’s with about 600,000 being sold. The RP06 disk drives were also very common and, while they were very large, each one held only about 200MB. The media was a removable pack and was often switched out to support different programs or data sets. They were also quite frequently used as a method of backup by copying data to another disk that had no data on it (“scratch disk”) and then moved offsite for disaster recovery.
This disk media pack had 10 platters that were each separated by about ½” of space and is where the disk heads would move into once the pack was inserted into the drive. The read/write heads would fly on a cushion of air over the platter surfaces and access data. This wasn’t perfect technology of course, and the heads would sometimes “touch down” on the media, damaging the surface of the disk, the heads, or both and would need to be replaced.
Back to my story.
This client was a family-owned company that worked with avionics and small aircraft repair and had logged a call that one of their disk drives was having issues. Another engineer wanted to come along with me, so we headed out to see the customer.
Now the mistake, or series of mistakes rather, that form foundation of my flashbulb moment.
Modern computer hardware has many built in data points that report back to the operating system that can be used to diagnose any issues. At this point in time however, that wasn’t true. To figure out what was going on, we would shut down the operating system and boot up to a diagnostic software that had specialty programs we could use to exercise the hardware and gather specific error information. This meant that we needed to have a scratch disk if we wanted to do any reading or writing to disk media and all clients kept several of these around for maintenance.
After running the diagnostics, we determined that the read/write heads in the disk needed to be replaced and we started into that task. I took a couple of hours to replace the heads and get them aligned properly, and then to test that everything worked ok, we would do some writing on the scratch disk.
When I asked for a scratch disk, the client said that he was out of them due to the problems they had been experiencing and that he would give us a backup disk, but to please be careful because it was the only backup disk he had. I told him that we wouldn’t write on it but only use it for reading of data to test the head alignment.
While the other engineer is going about this last task, I continued talking with the client. Then I went back into the computer room and assisted my co-worker in putting the client operating system and data disks back for them to use.
As I walked into the room, fear and anxiety hit me like a 2×4 across the forehead as I realized that the other guy had just run random data patterns all over the client’s backup disk! His primary disks were damaged from the bad heads and we just wiped out his only backup!
As the tightening in my stomach did its best to hold back the stress induced nausea, I went to tell the owner that we had just wiped out their data.
I explained to one of the owners (the husband) what happened. He was calm and immediately started figuring out a solution to the situation. He had about a six-month old backup disk in a bank vault, and they could re-enter the last six months of data. As he went about his thought process, his only focus was a solution to the problem. He was the introvert that became quiet in his thoughts and sought out the resolution in his mind.
I went back to the computer room and his wife then came in about sixty seconds later.
She was not an introvert.
Her position was that of the drill sergeant that I described earlier. The volume of her screams belied the tiny body that it came from. The top of her head came no higher than my chin, but face pointed up at mine and just inches apart she started in.
DID YOU REALLY JUST DELETE OUR DATA?!
TELL ME EXACTLY WHY YOU DELETED OUR DATA?!
HOW CAN ANYONE BE AS STUPID AS YOU?!
ARE YOU THE MOST STUPID PERSON IN THE WHOLE ENTIRE WORLD?!
My first response in my head was – hey – this isn’t my fault! The other guy did it! But there was nothing I could say, nor was this the time. The best thing I could do is take my punishment, let her vent as long as she needed and create in my mind a flashbulb moment that I still remember clearly almost 40 years later.
As soon are she had entered the room, my co-worker had quickly and conveniently left.
The chicken.
End of the story is that they eventually recovered. The six-month old backup disk was used, and they re-entered the data, but it wasn’t pretty, easy or cheap.
It left an indelible mark on my psyche of never wanting to be in that situation, nor put any of my clients in that type of situation ever again. I’ve spent much of my career in helping clients manage their data and I always think about what could possibly go wrong and how it might end up with data loss.
As I mentioned earlier, I wanted to say that it wasn’t my fault, but in fact, I knew it was.
Rarely do serious problems happen where just one thing go wrong. It’s almost always a combination of multiple failures along the way.
The problem with the disk heads had been going on for a while and had damaged several of their spare disks. If they had let us know about the problem earlier, they would have had several disks available, and they should have had better backup practices in place than what they did.
I never should have left the other engineer in the room doing the task by himself. The client warned me that this was his last disk, and I should have treated it with the value that it really had.
The other engineer should have taken the same viewpoint and been extra careful.
But each of these steps were missed, and because of it, valuable data, time, money and confidence were all lost. It was a learning experience that I will never forget and through it I learned to treat all of our client’s data like gold.
…
Since we started Dataedge, we have helped may clients manage their data and computer infrastructure and we always treat those opportunities like gold. We build business resilience into our designs by focusing on reliability, security and redundancy. We would value the opportunity to do this for your business and promise that we will treat your business data with the same value that you do.
Tim Hollingshead
www.dataedge.com
tim@dataedge.com
