The CPT: The Most Overlooked Threat to Your Data

How well protected is your data from all types of threats?  Is your data safe because it is backed up and mirrored?  Have you invested in significant hardware and network redundancies and even multi-million dollar remote sites?  What if I told you that your data is most likely still susceptible to a serious threat that nearly all businesses I interact with have overlooked.  What is it?

Crazy People.

Crazy people can happen anywhere at any time.  They can cause millions or even billions of dollars in damages in company assets and, sadly, even human life.  For this purposes of this blog, I will be focusing on the Crazy People Threat (CPT) to your data, why most companies are increasingly vulnerable and how you may be able to better protect your company and your job from such threats.

First, let’s define what I mean by a CPT.  A CPT, in this context, is any threat to your data caused by the irrational motivations of others.  They range from terrorists motivated by corrupted religious beliefs, disgruntled employees full of “boss hate”, chemical imbalances, rage control issues, drug abuse (now including caffeine overload), greed and a variety of other explainable and unexplainable causes.

As we engineer our applications, servers, networks and storage systems for maximum availability we go to great lengths to ensure redundancies at multiple levels.  We try to implement failure boundaries to prevent any one component of a system from compromising the entire system.  We even go so far as to build complete, geographically redundant sites to protect from the extremely rare, but serious threat of natural and unnatural disasters.

So why are we still vulnerable?  Because the threat comes from the last place we would expect.  Storage and Backup Administrators and Architects devote their talents and their lives to building high quality infrastructures to meet the needs of the business.  Many times they work all hours of the night and day to ensure internal and external customers have a positive experience with technology.  They are, or at least should be, some of the most highly respected and valued members of our companies.  It is for this reason that we often give them seemingly god-like power to turn their ideas into reality.  But, with the power to create, also comes the power to destroy.

Administrators, although they sometimes seem like more than people, the are-in fact-people too.  This “‘limitation” exposes them, like everyone else to the possibility of craziness.  One could make the argument that the draw of power even appeals to many administrators.  This in combination with the unnatural stresses and sometimes…um…unique…personalities of administrators, could make them more likely to turn crazy than the “normal” populace.

Unfortunately, craziness is very hard to detect, which is why neighbors always say things like “she was a nice person” or “he seemed completely normal.” It is for this reason that we must protect our businesses regardless of how much we respect and trust our admins.  I know you may have complete faith in your staff to stay loyal to the company’s interests, but consider this.  Let’s say you have invested hundreds of thousands or even millions of dollars in a fully protected disaster recovery site in the rare case of a natural disaster.  Now, watch your local nightly news and honestly weigh which has a greater possibility of occurring–a natural disaster or a CPT?

Please don’t misunderstand the premise of this post.  There are real and serious disorders that people struggle with and the last thing I want to do is make light of that.  As a business and a person we have a responsibility to look out for those around us and provide help to those who need it with wellness programs, concerned listening, helping hands, etc..  Sometimes life’s circumstances are hard and we need to be there for each other.  However,  the threat is real and while we have the best wishes for those who are ill, we also must think logically and protect our businesses from the possibility.

Two-Key Data Protection

The best way two assess your vulnerability to a CPT is with something I call “Two-Key Data Protection.”  In a typical enterprise environment you should have at least two copies of all data (sometimes more).  I call these copies the “Primary” and “Backup”.  You may have seen a movie where the president and some high-ranking military official decide to initiate a nuclear launch sequence.  During these scenes there are typically two keys that must be turned at the same time by both individuals or two key cards with codes that must be verified between them before the sequence can be initiated.  Likewise, no one person in your organization should have the power to destroy both primary and backup copies of data.  What are some scenarios where this might be possible?

• Backup Administrator and Storage Administrator are the same person/team
• Storage Administrators provision and control both primary storage and storage used for backup targets
• Storage replication is the only means of data protection or deletion of storage array snapshots can be replicated to mirror site

There are many other possibilities.  The main idea to keep in mind is that any situation where a single person can destroy all copies of data is a CPT vulnerability.

Why are businesses increasingly vulnerable?  A few reasons, really.

The first is infrastructure modernization.  Important and much-needed technologies (such as virtual machines, backup to disk, array-based data protection, etc..) are being implemented to add flexibility and speed to dated IT infrastructures. However, that same speed and flexibility is also given to destructive capabilities.  In dated, tape-based data protection schemes, tapes are usually managed with separate software by separate administrators on separate infrastructures.  In todays highly virtualized–and staff reduced–environments, storage and backup functions are often being combined into the same team.

Another reason has to do with speed of access.  The great thing about disk based backup approaches is that, unlike tape, data can be accessed very quickly for both backups and restores.  However, data that can be access quickly can also be destroyed quickly.  It is far easier to wipe a disk array than tapes in a library–or in a cave for that matter.

So how can you protect yourself?

Just be aware.  Try to implement policies and use Roll-Based-Access-Control (RBAC) to restrict permissions between individuals or teams for primary and backup data.  You will have to approach the subject gently as admins don’t necessarily like having their permissions reduced, especially due to suspicion of craziness!  But if it is a good policy for the business, I’m sure they can be persuaded to understand. If you are using array-based data protection only, ask you storage vendor for ways to prevent data on the backup array from being compromised in the event it is compromised on the primary array.  You may consider keeping tape around for less frequent, but longer term copies of data to prevent a security compromise that may take out both online copies from also taking out the tape copy.  Offline data is not easily compromised by online security failures.

I hope this has maybe raised this issue as a concern.  Following the Two-Key Data Protection best practice can not only protect you from the CPT, but there are countless human errors that can also be turned catastrophic if permissions allow it.  If you are still not convinced, consider the following:

http://www.firstcoastnews.com/news/news-article.aspx?storyid=100625