Wednesday, June 18, 2008

Kneel Before SOD

Separation Of Duties, or SOD is an area of focus in my organization. We have policies in place that forbid individuals from having write access to multiple computer environments in my company. That is, software developers have full access to the development environment, but they do not have unabated access to the QA or Production environments.
The reasoning behind this system is to prevent a rogue individual from trying to sabotage our production environments or hold it hostage like some bonehead sysadmin did to Medco a while back.
Companies should be prepared for this type of damage without worrying about whether their employees are going to try to blackmail them. It will probably happen through good old fashioned incompetence before that type of scenario anyway. The moral of the story is to be prepared to deal with adversity. If your data is valuable, make sure that it can be restored if something should happen to it.
Back to the policy. I'm not sure I'm sold on it.
Having a secure production environment is important, and allowing only competent disciplined and trustworthy people to access it is the key. Easier said than done, but we do our best.
The Segregation Of Duties approach to securing a production environment tries to minimize the number of people who have the ability to affect the production environment. In addition to limiting the people who have access strict rules requiring a traceable set of approvals is required to promote any change into the production environment.
The auditors love this system. For every file in the production environment they could, if they wanted to, follow a paper trail showing who approved putting it out there. That's the theory.
Most developers work with the understanding that any defect they put into production is going to be very painful to rectify. As a result they tend to do their best to be responsible despite considerable pressure to deliver.
So what happens when the pressure to deliver conflicts with the pressure to be in compliance? Well, how much pressure is there to be in compliance? Well, the people who want things done are paying money, the people who are interested in compliance aren't.
There are policies in place and there are people who check the systems to make sure that people don't do anything egregious, there are gaps in the policies though. People outside the inner sanctum of the trusted few who know the production passwords learn the production passwords. It's going to happen.
When it comes down to it, if a person knows that there is something simple that they can quietly do themselves to avert a mess, there will be people who will violate protocol to avert the mess.
There will be some who do it by the book. But there are some who will will assess the risk of getting caught and the pain of going through the lengthy protocol and conclude that simply fixing the problem is the better choice.
Is it a bad thing? Yes and no. It's a risky proposition to have individuals affecting a production environment. It's also an expensive proposition to have individuals run through an elaborate process that involves several layers of management to make a simple fix.
The risk of allowing individuals to make changes into a production environment directly is that every change could introduce additional problems. I worked for a company where one of the developers treated production as his own personal development environment. Following every time he or his team would make a release to production there would be a series of 'point releases' or emergency patches that would ensue for a few days. That company enforced absolutely no barriers in place to prevent this code cowboy from promoting his next best guess into production. The company incurred a tremendous amount of technical debt because of the last minute shenanigans.
The risk of creating an onerous set of processes to effect change is the process will impede an organization's ability to make changes when they need to.
Process and discipline are the two part epoxy that holds a Separation of Duties system together. Without a process, the separation of duties provides negative value to an organization. In the case of my previous company there technically was a separation of duties. Without discipline, the process is worthless because it will be abandoned as soon as it becomes inconvenient.
The system at my previous company actually did have a separation of duties system in place. It broke down because the code cowboy could walk over to his favorite system administrator and say 'I need you to fix this now' and the sysadmin would acquiesce his request without question. There was no defined process in place there, there also was no discipline in place for the system administrator or the developer.
As a result, the codebase at that company could consider a compliment from an adjective like spaghetti and it's application from a description of pogo-stickesque stability.

No comments: